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EXECUTIVE  SUMMARY 


Human  Systems  Integration  (HSI)  is  typieally  defined  by  seven  domains:  manpower, 
personnel,  training,  human  faetors  engineering,  habitability,  personnel  survivability,  and  safety 
and  oeeupational  health.  As  a  part  of  Department  of  Defense  aequisition  proeesses,  HSI  ensures 
that  operator,  maintainer  and  sustainer  eonsiderations  are  ineorporated  into  military  system 
designs.  Depending  on  faetors  sueh  as  eost,  sehedule  or  desired  system  performanee,  it  may  be 
neeessary  to  eompromise  on  design  features  that  ean  impaet  human  performanee,  whieh  results 
in  tradeoffs  within  and  between  HSI  domains. 

The  objeetive  of  this  researeh  was  to  eharaeterize  the  naturalistie  deeision  making  proeess 
used  by  HSI  praetitioners  to  eonduet  domain  tradeoffs  for  Naval  Aviation  aequisitions.  This 
report  deseribes  the  first  phase  of  a  proposed  two  phase  study  on  HSI  Tradeoff  Deeision  Making. 
Five  Critieal  Deeision  Method  (CDM)  interviews  were  eondueted  with  Subjeet  Matter  Experts 
(SMEs)  to  identify  (a)  the  primary  aetivities  eondueted  during  HSI  tradeoffs,  (b)  the  types  of 
faetors  that  affeet  tradeoff  deeision  making  proeesses,  and  (e)  the  types  of  knowledge  and 
experienee  needed  to  eonduet  tradeoff  analyses.  The  interview  data  were  analyzed  using  eontent 
analysis  methods  with  an  inter-rater  reliability  ending  study;  the  data  were  also  used  to  ereate 
deeision  requirements  tables,  whieh  were  validated  through  a  seeond  round  of  interviews  with 
three  SMEs. 

The  five  ease  studies  reported  in  the  CDM  interviews  related  to  Aequisition  Category 
(AC AT)  I  programs:  two  were  for  aireraft  platforms,  and  three  eoneerned  subsystems  and  the 
proeesses  required  to  operate  or  maintain  those  subsystems.  Two  ease  studies  foeused  on 
manpower/workload,  and  the  relevant  HSI  domains  were  manpower  and  HFE.  Three  ease 
studies  involved  hardware/software  (HW/SW)  design  ehanges;  the  relevant  HSI  domains  were 
HFE,  safety  &  oeeupational  health,  and  training. 

The  data  from  the  transeripts  deseribe  four  general  HSI  tradeoff  tasks:  (1)  evaluating  a 
eurrent  or  proposed  design  for  human  performanee  impaet,  (2)  evaluating  the  implieations  of  a 
diseovered  human  performanee  impaet  on  HW/SW  design,  performanee,  eost  and  sehedule,  (3) 
evaluating  any  potential  eost,  sehedule  and  performanee  impaets  on  other  teehnieal  diseiplines, 
sueh  as  systems  engineering,  test  and  evaluation,  and  logisties  (supportability)  impaet,  and  (4) 
evaluating  the  eost  and  sehedule  impaets  on  the  aequisition  program. 

The  deeision  requirements  tables  further  elaborate  on  these  HSI  Tradeoff  tasks.  Seven 
deeision  aetivities  were  identified,  with  deseriptions  of  (a)  the  eues  that  trigger  eaeh  aetivity,  (b) 
the  faetors  that  influenee  eaeh  aetivity,  (e)  the  typieal  judgments  made  in  the  eourse  of  eaeh 
aetivity,  (d)  the  typieal  ehallenges  faeed  in  the  eourse  of  eaeh  aetivity,  (e)  the  strategies  used  to 
deal  with  the  ehallenges. 

The  eontent  analysis  identified  four  primary  eognitive  aetivities  assoeiated  with  HSI 
tradeoff  deeision  making:  (1)  sensemaking  and  situation  assessment,  (2)  planning/adapting/ 
replanning,  (3)  uneertainty  and  risk  management,  and  (4)  using  opportunities  and  leverage 
points.  HSI  praetitioners  are  performing  teehnieal,  proeess,  risk  and  impaet  evaluations  during 
tradeoff  deeisions.  They  are  determining  when  and  how  to  leverage  previous  work.  They  are 
planning  and  replanning  proeesses  for  analysis,  testing  and  implementation.  They  take  the  time 
to  evaluate  eurrent  eireumstanees  against  previously  experieneed  analogous  situations.  They 
also  interfaee  with  IPT  members,  stakeholders  in  other  organizations,  and  individuals  they 
worked  with  in  the  past  to  gather  and  proeess  information  and  data. 
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Particularly  for  the  HW/SW  cases,  considerable  effort  is  made  to  put  together  a  cost, 
schedule  and  performance  argument  to  present  to  program  management.  Successfully  generating 
this  argument  requires  additional  time  and  resources  to  gather  and  evaluate  information  on 
contractor  work  plans,  available  funding,  cost  estimates  (expenditures  and  savings)  and 
quantitative  and  qualitative  HW/SW  and  personnel  performance  impaets.  This  further  supports 
the  need  for  additional  researeh  and  development  efforts  on  HSI  deeision  support  tools. 

Further  researeh  is  also  needed  to  identify  speeifie  HSI  tradeoff  KSAs,  and  determine  whieh 
KSAs  may  require  additional  workforee  training. 
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1.0  INTRODUCTION 

Department  of  Defense  (DoD)  Instruetion  5000.02,  Operation  of  the  Defense  Aequisition 
System,  Enelosure  8  (2008)  requires  program  managers  to  develop  a  Human  Systems  Integration 
(HSI)  plan  in  order  to  .optimize  total  system  performanee,  minimize  total  ownership  eosts, 
and  ensure  that  the  system  is  built  to  aeeommodate  the  eharaeteristies  of  the  user  population  that 
will  operate,  maintain,  and  support  the  system”  (p  60).  HSI  addresses  seven  domains  in  support 
of  human  eentered  systems  design:  manpower,  personnel,  training.  Human  Faetors  Engineering 
(HFE),  habitability,  personnel  survivability,  and  safety  &  oeeupational  health. 

Depending  on  the  teehnieal  seope  of  the  system  to  be  designed,  there  is  typieally  at  least 
one  HSI  representative,  or  a  representative  from  any  of  the  seven  HSI  domains,  serving  on  an 
aequisition  program’s  Integrated  Produet  Team  (IPX).  It  is  the  responsibility  of  the  designated 
HSI  praetitioner(s)  to  eoordinate  with  other  IPX  members  to  ensure  that  over  the  system’s 
lifeeyele:  (a)  there  is  adequate  manpower/manning,  (b)  the  designated  personnel  will  have  the 
right  skill  sets  and  will  be  adequately  trained,  and  (e)  the  system’s  hardware,  software  and 
proeesses  are  designed  for  safe  and  effieient  use. 


1.1  What  is  an  HSI  tradeoff? 

Malone,  Pharmer,  Eoekett-Reynolds,  &  Duma  (2008)  deseribe  “tradeoffs  that  must  be 
eondueted  among  the  HSI  domains  to  aehieve  the  most  effeetive  and  affordable  integration  of  the 
human  element  with  system  hardware,  software,  firmware,  eourseware,  proeedures, 
organizations,  environments,  and  information”  (p  1954).  Typieally,  the  system  design  features 
under  eonsideration  will  dietate  whieh  HSI  domains  are  relevant  to  a  given  tradeoff  analysis.  For 
example,  Chapanis  (1996)  deseribes  the  following  tradeoff  seenario,  whieh  illustrates  the  human 
impaets  that  need  to  be  eonsidered  in  addition  to  hardware  and  software  performanee: 

“...suppose  Funetion  A  ean  be  performed  faster  on  Computer  System  X,  but 
Funetion  B  ean  be  performed  faster  on  Computer  System  Y. .  .How  do  you  strike  a 
balanee  between  the  savings  in  time  versus  the  eost  of  errors?  Does  inereased 
operator  eomfort  inerease  produetivity  and,  if  it  does,  how  ean  that  be  translated 
into  dollar  savings?  Sinee  more  highly  seleeted  personnel  require  less  training,  is 
it  better  to  spend  more  money  on  seleetion  or  on  training”  (p  283). 

From  a  teehnieal  standpoint,  the  level  of  eoordination,  data  eolleetion  and  data  analysis 
required  to  aeeomplish  these  tasks  will  depend  on  the  system’s  requirements  and  level  of 
maturity  (e.g.  teehnology  readiness  levels).  From  a  programmatie  standpoint,  tradeoffs  will  also 
be  impaeted  by  the  aequisition  program’s  budget  and  sehedule.  However,  during  the  eourse  of 
an  aequisition  program,  any  of  these  teehnieal  and  programmatie  faetors  may  ehange,  resulting 
in  a  need  to  reevaluate  the  eost,  sehedule  or  performanee  of  the  overall  system  and/or  its 
eomponents.  From  an  HSI  standpoint,  this  ean  result  in  tradeoffs  within  and  between  the  HSI 
domains  that  impaet  not  only  human  performanee,  but  also  that  of  operational,  maintenanee  and 
support  proeedures. 
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1.2  Research  in  HSI  Tradeoffs 

Recent  research  in  HSI  tradeoffs  from  both  the  Naval  Postgraduate  School  and  the 
Massachusetts  Institute  of  Technology  have  centered  on  decision  support  methods  and  tools. 
Simpson  (2006)  proposed  a  tool  with  three  interfaces  (an  HSI  Resource  Search  interface,  a 
parameter  interaction  editor,  and  an  HSI  Trade  Space  Tool),  which  used  levels  of  manpower, 
personnel,  training  and  human  engineering  as  inputs,  and  risk,  cost,  schedule  and  performance  as 
outputs.  Lazzaretti  (2008)  leveraged  this  research  and  the  HSI  Trade  Space  tool  to  evaluate 
operational  readiness  and  safety  as  a  function  of  manning  levels  using  historical  frigate  data. 

Cunio  and  Cummings  (2009)  describe  a  downselection  decision  support  aid  called  the 
Systems  Integration  Tool  for  HSI  Evaluation  (SITHE),  which  was  designed  to  help  decision 
makers  choose  the  best  tools  to  evaluate  HSI  for  a  system  of  interest.  Coley  (2010)  investigated 
the  impact  of  the  visualization  techniques  used  in  downselection  tools  on  decision  maker 
behaviors  when  using  these  tools. 

Higgins  and  Mack  (2006)  used  the  responses  from  an  alternatives  analyses  survey  tool 
called  ABRAHAM  to  identify  HSI  areas  of  high  risk  to  equipment  operators,  maintainers  and 
supporters.  Desmond  (2007)  used  the  Quantified  Judgment  Model  to  compare  the  combat 
potential  of  two  Marine  Distributed  Operations  units,  using  Doctrine,  Organization,  Training, 
Materiel,  Eeadership  and  education.  Personnel,  Facilities  (DOTMEPF)  data.  Finally,  Fiu  (2010) 
investigated  the  impact  of  integrating  HSI  into  an  existing  systems  engineering  cost  model,  to 
better  incorporate  HSI  in  acquisition  program  planning. 


1.3  HSI  Tradeoff  Decisions  as  a  Naturalistic  Decision  Making  Process 

As  demonstrated  by  previous  research  efforts,  HSI  tradeoff  analyses  include  evaluations 
of  the  impact  that  different  design  features  will  have  on  hardware,  software,  and  human 
performance.  They  also  include  making  decisions  or  choices  to  select  one  design  feature  over 
another,  considering  other  factors  such  as  cost  and  schedule.  However,  conducting  HSI  tradeoffs 
is  much  more  than  a  simple  choice  among  alternatives. 

As  a  sociotechnical  system,  the  DoD  integrated  defense  acquisition,  technology,  and 
logistics  life  cycle  management  system  is  influenced  by  various  internal  and  external 
environmental  factors  such  as  timing  and  duration  of  each  acquisition  phase,  IPT  personnel 
changes,  military  policy,  and  Congressional  funding.  Decisions  typically  take  place  in  working 
level  IPTs,  through  collaboration  between  a  variety  of  technical  and  management  stakeholders. 
These  decisions  are  facilitated  by  a  high  degree  of  deliberation  in  an  environment  influenced  by 
technical,  programmatic  and  organizational  factors.  Arguably,  these  characteristics  of  tradeoff 
decisions  align  with  the  eight  factors  that  characterize  decision  making  in  a  naturalistic  setting 
(Orasanu  and  Connelly,  1993): 

1 .  Ill  Structured  problems 

2.  Uncertain  dynamic  environments 

3 .  Shifting,  ill-defined,  or  competing  goals 

4.  Action/Feedback  Foops 

5.  Time  Stress 

6.  High  Stakes 

7.  Multiple  Players 

8 .  Organizational  Goals  and  N orms 
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Naturalistic  Decision  Making  (NDM)  studies  have  been  conducted  using  Subject  Matter 
Experts  (SMEs)  in  a  variety  of  work  domains,  including  military  operations  (Bisantz,  et  al,  2003; 
Cohen,  Freeman  &  Thompson,  1996;  Phillips,  et  ah,  2001;  Stanton,  et  ah,  2006),  medical 
operations  (Fackler,  et  al,  2009;  Patterson,  Woods,  Cook  &  Render,  2007;  Wong,  2004),  system 
development  (Hoffman,  Neville,  &  Fowlkes,  2009;  Zannier,  Chiasson,  &  Maurer,  2007), 
information  technology  project  management  (Taylor,  2007),  and  airline  safety  (Macrae,  2009). 
Cognitive  task  analysis  methods  have  been  used  to  study  NDM  in  a  variety  of  work  domains. 
Combinations  of  techniques  such  as  Critical  Decision  Method  (CDM)  interviews,  questionnaires, 
think  aloud  protocols  and  direct  observations  have  been  used  to  collect  data  on  individual  and 
team  decision  making  (Crandall,  Klein,  &  Hoffman,  2006). 


1.4  Research  Objectives  and  Goals 

The  objective  of  this  research  was  to  empirically  investigate  the  NDM  processes  used  by 
HSI  professionals  on  Naval  Aviation  acquisition  programs  to  assess  cost,  schedule  and 
performance  tradeoffs  within  and  between  HSI  domains.  The  goals  of  this  research  were  to 
investigate  HSI  tradeoff  decision  making  CDM  and  decision  requirements  table  interviews  with 
HSI  SMEs.  The  original  scope  of  this  research  also  included  a  plan  to  conduct  a  computer  based 
decision  making  assessment  study  using  inputs  from  both  novices  and  experts.  However,  due  to 
project  resource  constraints,  only  the  CDM  and  decision  table  interviews  were  completed.  This 
report  documents  the  results  of  these  interviews. 

CDM  is  a  knowledge  elicitation  method  specifically  designed  to  facilitate  the  recollection 
of  decision  making  processes  used  in  specific  non-routine  or  difficult  incidents  (Klein, 
Calderwood,  &  MacGregor,  1989).  Participants  are  asked  to  recount  the  timeline  and  details  of 
the  incident  through  a  series  of  probe  questions  that  target  different  cognitive  aspects  of  the 
decision,  such  as  goals,  mental  modeling,  and  errors  made  or  avoided  (Hoffman,  Crandall,  & 
Shadbolt,  1998).  Depending  on  the  research  focus,  the  probe  questions  are  tailored  to  highlight 
specific  cognitive  processes  of  interest. 

As  described  by  Klein  (1998),  a  decision  requirements  exercise  enables  decision  makers 
“to  identify  (1)  key  judgments  and  decisions  facing  them,  (2)  why  they  are  difficult,  and  (3) 
where  they  can  go  wrong.  These  decision  requirements  are  the  high  drivers,  the  specific  decision 
skills  that  they  need  to  polish”  (p.  105).  The  results  of  CDM  interviews  can  be  used  facilitate  the 
creation  of  these  decision  tables.  For  example,  Wong  (2004)  generated  incident  summaries  and  a 
decision  chart  that  provided  the  necessary  information  to  produce  a  decision  analysis  table. 
Phillips,  McDermott,  Thordsen,  McCloskey,  &  Klein  (1998)  combined  findings  from  CDM, 
knowledge  audit  and  task  diagram  interviews  to  generate  initial  decision  requirements  tables  that 
were  later  validated  with  additional  SME  inputs  (Phillips,  et  ah,  2001). 

This  research  in  HSI  tradeoff  decision  making  facilitates  the  identification  of  knowledge, 
skills,  and  abilities,  which  is  a  critical  step  in  determining  (a)  which  tradeoff  decision  tasks  may 
require  additional  workforce  training  and,  (b)  which  tasks  are  candidates  for  automation  aids, 
and  are  worth  spending  additional  resources  to  develop  some  kind  of  decision  support  tool.  Such 
efforts  support  the  development  of  the  S&T  workforce,  as  well  as  enhanced  HSI  and  acquisition 
process  capabilities. 

As  described  in  the  2009  Naval  Science  and  Technology  (S&T)  plan,  the  primary  drivers 
for  total  ownership  cost  are  acquisition  of  platforms  and  systems,  maintenance  and  life-cycle, 
and  manpower.  Because  HSI  domain  tradeoff  analyses  impact  each  of  these  cost  drivers,  it  is 
beneficial  for  the  Naval  Aviation  Enterprise  (NAE)/Navy  to  continuously  improve  the  HSI 
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methodologies  used  to  perform  sueh  analyses  throughout  the  aequisition  lifeeyele.  More 
aeeurate  human  related  eost,  sehedule  and  performanee  tradeoffs  will  not  only  impaet  the  quality 
of  manpower  optimization  and  training  effeetiveness  efforts,  but  will  also  inerease  system  and 
platform  affordability  and  availability. 


2.0  METHOD 

For  the  CDM  and  deeision  requirements  table  interviews,  approval  from  the  Institutional 
Review  Boards  at  both  the  Naval  Air  Warfare  Center  Training  Systems  Division  (NAWCTSD) 
and  the  Naval  Air  Warfare  Center  Aireraft  Division  (NAWCAD)  Patuxent  River  was  obtained. 


2.1  Participants 

This  researeh  required  the  partieipation  of  HSI  SMEs  with  at  least  eight  years  of 
experienee  in  Naval  Aviation  aequisition,  and  at  least  five  years  of  experienee  speeifieally 
eondueting  aviation  related  HSI  tradeoff  analyses.  Potential  partieipants  for  this  study  were 
identified  by  AIR  4.6  management.  Eaeh  individual  was  then  reeruited  via  email  request  from 
the  study’s  prineipal  investigator  to  voluntarily  partieipate  in  this  study. 

Five  CDM  interviews  were  eondueted  over  the  eourse  of  three  months.  Three  of  these 
five  individuals  also  partieipated  in  the  deeision  requirements  table  interviews,  whieh  were 
eondueted  over  a  two  month  period.  Table  1  summarizes  the  demographie  profiles  of  the  CDM 
interview  partieipants. 


Table  1:  CDM  Interview  Participant  Demographic  Profiles 


Educational  background 

Psychology  (Four  participants) 

Engineering  (One  participant) 

Years  of  DoD  acquisition  experience 

Range:  8-28  years 

Years  of  Naval  Aviation  work  experience 

Range:  4-28  years 

Years  of  HSI 

Range:  5-27  years 

Years  of  experience  on  the  specific  acquisition 
program  when  the  decision  occurred 

Range:  6  months  -  3  years 

Years  of  total  acquisition  experience  when  the 
decision  occurred? 

Range:  4-18  years 

2.2  Critical  Decision  Method  Interviews 

For  this  researeh,  struetured  eritieal  deeision  method  (Hoffman,  et  ah,  1998;  O’Hare, 
Wiggins,  Williams,  &  Wong,  1998;  Taylor,  2007)  interview  questions  were  used.  The  questions 
were  peer  reviewed  before  subjeet  reeruitment  began.  The  interview  protoeol  and  a  list  of  the 
demographies  and  probe  questions  are  provided  in  Appendix  A. 


2.2.1  Data  Collection 

Signed  informed  eonsent  forms  were  obtained  from  all  study  partieipants.  All  interviews 
were  reeorded  to  support  data  transeription,  and  were  eondueted  in  a  elosed  door  eonferenee 
room  to  better  eontrol  ambient  noise  for  the  reeording  equipment.  Eaeh  interview  lasted 
approximately  1.5-2  hours.  All  interviews  were  unelassified. 
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SMEs  were  first  asked  demographie  questions  (e.g.  edueational  baekground,  years  of 
aequisition  experienee,  ete).  Next,  they  were  asked  to  provide  a  high  level  deseription  of  an  HSI 
tradeoff  that  oeeurred  within  the  last  five  years,  and  deseribe  the  details  about  that  ineident. 
Probe  questions  eentered  on  faetors  like  the  timeline  of  events,  interaetions  between  personnel, 
planning  aetivities,  analogous  situations,  mental  modeling,  and  the  use  of  experienee. 


2.2.2  Data  Analysis 

In  the  transeription  of  the  audio  files  generated  from  the  interviews,  the  anonymity  of 
eaeh  partieipant  was  proteeted  by  (1)  replaeing  subjeet  names  with  a  partieipant  number,  (2) 
replaeing  speeifie  aequisition  program  names  with  its  designated  Aequisition  Category  (AC AT) 
level,  whieh  eategorizes  a  program  by  expenditure  amount,  and  (3)  replaeing  the  names  of  other 
personnel  mentioned  with  their  job  title  or  a  similar  deseriptor. 

After  the  audiotapes  were  transeribed,  a  eontent  analysis  for  eoding  and  emergent  themes 
was  performed  (Krippendorff,  2004;  Neale  &  Niehols,  2001;  Neuendorf,  2002).  All  four  eoders 
who  partieipated  in  the  analysis  were  at  least  Defense  Aequisition  Workforee  Improvement  Aet 
(DAWIA)  Level  I  eertified  in  Systems  Engineering  and  had  either  a  degree  and/or  relevant  work 
experienee  in  Human  Faetors. 

All  of  the  data  in  the  transeripts  exeept  the  responses  to  the  “Experienee”  and  “What -If  ’ 
questions  were  used  in  the  eoding  analysis;  the  “Experienee”  and  “What -If  ’  responses  were 
analyzed  separately.  The  eodes  used  in  this  researeh  were  based  on  the  maeroeognition 
funetions  and  proeesses  eommonly  found  in  eognitive  task  analysis  researeh,  deseribed  by 
Crandall,  et  al.  (2006)  and  Klein  (1998)  as  “the  eolleetion  of  eognitive  proeesses  and  funetions 
that  eharaeterize  how  people  think  in  natural  settings”  (p  136).  These  are  higher  level  eognitive 
proeesses  (e.g.  managing  attention  by  determining  what  tasks  to  foeus  on),  as  opposed  to 
mieroeognitive  proeesses  (e.g.  serial  vs.  parallel  attention  proeessing). 

An  inter-rater  reliability  study  was  eompleted  in  three  phases.  In  the  first  phase,  two 
transeripts  were  used  to  reduee  the  total  number  of  maeroeognition  funetions  and  proeesses  to 
the  following  four  eodes: 

•  Sensemaking  and  Situation  Assessment 

■  Did  someone  perceive  or  evaluate  information? 

■  Did  someone  diagnose  a  situation? 

■  Did  someone  anticipate  how  a  situation  might  develop  in  the  future? 

•  Planning/  Adapting/Replanning 

■  Did  someone  create  a  strategy  to  complete  specific  activities  within  a 
certain  timeframe? 

■  Did  someone  modify,  adjust  or  replace  a  plan  that  was  already  being 
implemented? 

•  Uncertainty  and  risk  management 

■  Did  someone  not  know  or  understand  something  which  impacted  or 
would  impact  how  the  tradeoff  analysis  proceeded  (e.g.  critical  data  were 
missing  or  considered  unreliable,  goals  were  unclear,  problems  were  not 
clearly  stated,  or  people  were  not  sure  what  to  do  next )? 

•  Using  opportunities  and  leverage  points 

■  Did  someone  turn  an  opportunity  or  a  leverage  point  into  a  course  of 
action? 
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Klein  (1998)  describes  a  leverage  point  as  an  occurrence  that  can  impact  the  direction  of 
a  given  situation:  “Leverage  points  are  just  possibilities  -  pressure  points  that  might  lead  to 
something  useful,  or  might  go  nowhere  (pi  16)... Leverage  points  provide  fragmentary  action 
sequences,  kernel  ideas,  and  procedures  for  formulating  a  solution”  (pi  17). 

The  identification  of  these  four  codes  as  most  relevant  to  HSI  Tradeoffs  was 
accomplished  over  several  iterations,  which  were  facilitated  by  a  calculated  percent  agreement 
between  coders,  and  subsequent  consensus  building  discussions  to  further  refine  how  best  to 
interpret  the  text  selections  from  the  transcripts. 

To  calculate  rater  reliability  scores,  a  boundary  based  method  similar  to  Carletta,  et  al. 
(1997)  was  initially  used.  The  calculated  Kappa  score  using  this  method  reflects  the  agreement 
between  raters  on  the  number  of  times  the  codes  are  used  for  each  boundary.  However,  the 
resulting  Kappa  values  were  very  low  because  the  percent  error  was  very  close  to  the  percent 
agreement,  despite  having  over  30  boundaries  for  each  of  the  two  transcripts.  This  was 
attributed  to  the  fact  that  the  procedure  required  each  coder  to  review  every  text  selection  four 
times,  once  for  each  code,  and  respond  either  “Yes”  or  “No”  if  a  code  applied.  Fleiss’  Kappa 
(Fleiss,  1971;  Gwet,  2010)  was  then  used,  because  it  is  a  similar  method  of  assessing  agreement 
on  the  assignment  of  categorical  ratings.  However,  this  Kappa  was  also  very  low.  Therefore,  it 
was  decided  to  rely  primarily  on  only  percent  agreement  and  consensus.  Percent  agreement  was 
47%  after  the  first  coding  phase. 

The  second  coding  phase  was  performed  to  further  validate  the  use  of  four  codes,  using  a 
representative  data  set  that  included  71  text  selections  from  across  all  five  transcripts.  The  same 
“Yes/No”  procedure  was  used  to  categorize  text  selections.  After  the  second  phase,  percent 
agreement  for  each  code  was  over  60%. 

The  final  phase  was  performed  using  the  four  codes  on  229  text  selections  from  all  five 
transcripts.  This  phase  used  the  same  “Yes/No”  procedure.  Fifty  text  selections  were  not 
assigned  to  any  of  the  four  codes;  the  coders  determined  that  they  contained  repeated  and 
extraneous  information,  such  as  commentary  on  or  embellishment  of  already  stated  facts.  For 
the  remaining  179  segments,  more  than  one  code  could  be  assigned  to  each  text  selection  if 
deemed  applicable.  After  each  text  selection  was  coded,  emergent  themes  were  generated  within 
each  code;  a  text  selection  could  be  assigned  to  more  than  one  theme.  Percent  agreement  was 
not  calculated  in  this  phase;  only  consensus  was  used. 


2.3  Decision  Requirements  Table  Interviews 

In  parallel  with  the  final  coding,  decision  requirements  tables  were  created  using  the 
contents  of  the  same  five  transcripts.  Three  researchers  worked  on  the  sequential  lists  that  were 
used  to  create  the  initial  decision  tables,  following  the  method  described  by  Wong  (2004).  Two 
researchers  developed  the  initial  decision  requirements  tables  using  the  sequential  lists;  no 
coding  took  place,  only  team  consensus.  For  each  decision  activity  in  the  table,  there  is  a  list  of 
(a)  cues  that  trigger  the  activity,  (b)  factors  that  influence  the  activity,  (c)  typical  judgments 
made  in  the  course  of  the  activity,  (d)  typical  challenges  faced  in  the  course  of  the  activity,  (e) 
strategies  used  to  deal  with  the  challenges. 

The  same  potential  participants  that  were  identified  for  the  CDM  interviews  were 
recruited  via  email  to  participate  in  the  decision  table  verification  and  validation  interviews. 
Interview  questions  were  peer  reviewed  before  subjects  were  recruited  for  the  decision  table 
verification  and  validation  interviews.  Participants  were  asked  to  review  the  contents  of  the 
tables  and  recommend  corrections  and  edits.  SMEs  were  also  asked  to  describe  decision 
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characteristics  that  clearly  distinguish  experts  from  novices.  The  interview  protocol  for  decision 
table  verification  and  validation  is  provided  in  Appendix  A. 


2.3.1  Data  Collection  and  Analysis 

A  pilot  study  with  one  SME  was  eondueted  to  verify  the  eontents  of  the  tables  and  the 
appropriateness  of  the  interview  questions.  Signed  informed  eonsent  forms  were  obtained  from 
all  study  partieipants.  All  interviews  were  reeorded  to  support  data  transeription,  and  were 
eondueted  in  a  elosed  door  eonferenee  room  to  better  eontrol  ambient  noise  for  the  reeording 
equipment.  Eaeh  interview  lasted  approximately  1-1.5  hours.  All  interviews  were  unelassified. 

At  the  end  of  data  eolleetion,  the  interview  reeordings  were  transeribed  and  redueed  to 
update  the  initial  deeision  requirements  tables  with  the  inputs  from  the  SMEs.  A  summary  of  the 
identified  Noviee/Expert  differenees  was  also  ereated. 


3.0  RESULTS 

This  seetion  deseribes  the  key  findings  from  the  summarized  results  of  the  CDM  and 
deeision  table  interview  data. 


3.1  Case  Study  Summaries 

All  five  ease  studies  related  to  ACAT  I  programs:  two  were  for  aireraft  platforms,  and 
three  eoneerned  subsystems  and  the  proeesses  required  to  operate  or  maintain  those  subsystems. 
Two  ease  studies  foeused  on  manpower/workload,  and  the  relevant  HSI  domains  were 
manpower  and  HFE.  Three  ease  studies  involved  hardware/software  (HW/SW)  design  ehanges; 
the  relevant  HSI  domains  were  HFE,  safety  &  oeeupational  health,  and  training.  The 
summarized  responses  to  the  CDM  interview  probe  questions  ean  be  found  in  Appendix  B. 

The  manpower  ease  studies  were  not  tradeoffs  in  the  true  sense  of  the  word.  They  were 
more  of  an  evaluation  of  the  known  design  tradespaee,  to  verify  if  the  existing  manpower 
estimate  was  valid  for  the  design,  and  whether  or  not  the  design  added  workload.  However,  for 
one  of  these  ease  studies,  a  workload  impaet  was  diseovered.  As  a  result,  teehnieal  analysis  for 
an  appropriate  mitigation  and  a  eost  analysis  for  a  eontraet  modifieation  were  performed. 

The  HW/SW  tradeoffs  were  system  design  tradeoffs.  In  eaeh  ease,  evaluations  and 
analyses  were  performed  to  elearly  artieulate  the  impaet  of  a  partieular  design  feature  on  the 
operator’s  or  maintainor’s  ability  to  interaet  with  the  system  as  intended,  in  both  mission 
exeeution  and  emergeney  seenarios.  Potential  impaets  on  training  were  also  identified  in  two  of 
the  ease  studies. 

The  data  from  the  transeripts  deseribe  four  general  HSI  tradeoff  tasks:  (1)  evaluating  a 
eurrent  or  proposed  design  for  human  performanee  impaet,  (2)  evaluating  the  implieations  of  a 
diseovered  human  performanee  impaet  on  HW/SW  design,  performanee,  eost  and  sehedule,  (3) 
evaluating  any  potential  eost,  sehedule  and  performanee  impaets  on  other  teehnieal  diseiplines, 
sueh  as  systems  engineering,  test  and  evaluation,  and  logisties  (supportability)  impaet,  and  (4) 
evaluating  the  eost  and  sehedule  impaets  on  the  aequisition  program.  These  four  aetivities  may 
seem  sequential,  but  they  are  aetually  eoneurrent  as  depleted  in  Figure  1.  Not  only  ean  different 
IPT  members  work  on  eaeh  task  in  parallel,  but  eonditional  or  preliminary  analyses  with 
doeumented  assumptions  ean  be  performed  in  eaeh  area,  then  revised  as  additional  information 
beeomes  available. 
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Figure  1:  HSI  Tradeoff  Process 


3.2  Decision  Requirements  Tables 

The  decision  requirements  tables  further  elaborate  on  the  HSI  practitioners’  role  in  the 
HSI  Tradeoff  Process  with  specific  decision  activities.  Seven  decision  activities  were  identified, 
with  descriptions  of  (a)  the  cues  that  trigger  each  activity,  (b)  the  factors  that  influence  each 
activity,  (c)  the  typical  judgments  made  in  the  course  of  each  activity,  (d)  the  typical  challenges 
faced  in  the  course  of  each  activity,  (e)  the  strategies  used  to  deal  with  the  challenges.  The 
complete  decision  requirements  tables  can  be  found  in  Appendix  C.  The  seven  decision 
activities  are  as  follows: 

•  Workload/Manpower: 

1 .  Examine/  Investigate  initial  manpower  estimate  and  the  assumed 
workload  for  intended  system  design 

2.  Create  Program  Risk  for  Manpower 

3 .  Perform  analyses  to  confirm  workload  and  manpower  for  the 
intended  system  design 

•  Hardware/Software  (HW/SW)  Changes: 

1 .  Evaluate  initial  system  design  for  potential  problems 

2.  Follow-up  with  contractor  after  problem  identification 

3 .  Government  HFE(s)  (without  contractor)  explore/generate  design 
options  to  later  present  to  contractor,  Fleet,  IPT,  and/or  program 
office  (IPT  and  contractor  may  or  may  not  simultaneously 
investigate  other  design  options) 

4.  IPT  (including  contractor)  explores/generates  design  options  to 
present  to  the  program  office  [Iterative  process] 

Various  cues  such  as  Navy  or  DoD  policy,  emergent  Fleet  needs,  design  changes,  test 
results,  uncooperative  contractors,  or  changes  in  program  funding  or  resources  are  examples  of 
occurrences  that  prompted  a  need  to  perform  a  tradeoff  activity.  Examples  of  factors  that 
influenced  the  tradeoff  activities  include  the  availability  of  workload  or  task  data,  and  time. 
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money  and  resources  to  perform  analyses.  Finally,  the  types  of  judgments  that  HSI  practitioners 
make  during  the  course  of  a  tradeoff  decision  process  are  summarized  as  follows: 

•  Scope  of  Problem/Issue/Risk 

•  System  Characteristics 

•  Alignment  of  Manpower  Levels  and  System  Design 

•  Analysis  Process  Used 

•  Data  Quality  and  Reliability 

•  Adequacy  of  Analysis  Results/System  Redesign 

•  Program  Office  Impacts/Influences 

•  Contractor  Performance 

It  is  important  to  note  that  HSI  practitioners  are  doing  technical  evaluations  as  well  as 
process  evaluations.  Time  is  taken  to  judge  data  validity  and  accuracy,  including  reviews  of 
analysis  tools  and  methodologies  used  by  others  as  well  as  themselves.  Practitioners  reportedly 
leveraged  lessons  learned  from  personal  experience,  advice  from  other  practitioners  and 
available  information  from  past  acquisition  programs,  industry  and  the  research  literature.  These 
points  are  frirther  highlighted  in  the  content  analysis  results  presented  in  the  next  section. 

Finally,  HSI  practitioners  face  a  variety  of  technical  and  programmatic  challenges  that 
impact  the  course  of  the  tradeoff  decision  process  for  both  Manpower/Workload  and  HW/SW 
changes.  Table  2  summarizes  the  reported  challenges,  and  what  experts  said  they  are  doing  to 
deal  with  them.  It  is  very  apparent  that  collaboration,  coordination  and  discussion  are  key 
strategies  for  all  of  the  challenges. 


Table  2:  HSI  Tradeoff  Challenges  and  Strategies 


Challenges 

Strategies 

Working  with  a  conceptual 
system  design,  a  changing  system 
design,  and/or  changing  mission 
parameters 

•  Characterize  the  analysis  results  relative  to  what  is  known, 
understand  the  limitations  of  the  analysis,  document  assumptions 

•  Evaluate  data  collection  and  analysis  methods  used 

•  Coordinate  with  SMEs  for  the  most  current  information 

•  Document  system  design  as  a  risk  (if  appropriate) 

•  Resolve  through  further  analysis  or  testing 

•  Modify  the  contract  statement  of  work  (if  required) 

Questionable  data  accuracy  and 
reliability 

•  Evaluate  data  collection  and  analysis  methods  used 

•  Resolve  through  further  analysis  or  testing 

•  Gather  additional  SME  inputs 

•  Govemment/contractor  discussions  or  joint  analysis  efforts 

Questionable  requirements 
compliance  or  requirement 
applicability  to  the  design 

•  Consult  with  other  IPT  members  and  SMEs 

•  Resolve  compliance  issues  with  contractor  through  discussions 
(include  management  and  contracting  officer  deemed  necessary) 

•  Revise  the  requirements  (if  appropriate) 

Insufficient  funding  and/or 
staffing  to  perform  analyses 

•  Petition  for  additional  funding/resources 

•  Eeverage  data  from  previous  studies 

•  Document  as  a  risk  (if  appropriate) 

Disagreements  between  HFEs  and 
IPT  members,  program  office, 
and  other  stakeholders 

•  Discussions,  with  supporting  technical  evidence 

•  Rely  on  IPT  leadership  to  manage  team  performance  and  HFE 
leadership  to  help  resolve  conflicts 
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3.3  Content  Analysis  Results 

As  described  above,  two  main  tradeoff  categories  were  reported:  Manpower/Workload 
and  HW/SW  changes.  Figures  2  and  3  show  the  breakdown  of  these  categories  by  the  four 
macrocognition  functions  and  processes  (aka  cognitive  activities)  identified  by  the  content 
analysis.  Because  the  same  transcript  data  were  used  to  perform  the  content  analysis  and  create 
the  decision  tables,  the  text  selections  used  in  each  analysis  were  aligned  and  counted.  The 
percentages  in  the  figures  represent  the  number  of  overlaps,  which  reflects  the  levels  of  each 
cognitive  activity  used  in  each  type  of  tradeoff  A  summary  of  the  mapping  of  the  decision 
activities  to  the  cognitive  activities  can  be  found  in  Appendix  D. 


Manpower  Workload 

10% 

■  Sensemaking  and 

\  Situation  Assessment 

\  ■  Pianning/ Adapting/ 

/  24% 

\  Repianning 

52%  /  ■  Uncertainty  and  Risk 

/  Management 

\  14% 

/  ■  Using  Opportunities 

and  Leverage  Points 

Figure  2:  Manpower  Workload  Cognitive  Activities 


HW/SW  Changes 

■  Sensemaking  and 

/  17%  X 

Situation  Assessment 

■  Pianning/  Adapting/ 

14%  ^ 

Repianning 

\  / 

■  Uncertainty  and  Risk 

Management 

\  16%  / 

■  Using  Opportunities 

and  Leverage  Points 

Figure  3:  HW/SW  Changes  Cognitive  Activities 


Table  3  lists  the  frequency  counts  of  segments  for  each  cognitive  activity  from  the 
content  analysis.  Because  more  than  one  code  could  be  assigned  to  each  text  selection,  and  a 
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text  selection  could  be  assigned  to  more  than  one  theme,  the  total  frequency  count  is  higher  than 
the  number  of  text  segments. 


Table  3:  Frequency  counts  of  segments  for  each  code 


Sensemaking  and 
Situation 
Assessment 

Planning/ 

Adapting/ 

Replanning 

Uncertainty  and 
risk 

management 

Using  opportunities 
and 

leverage  points 

TOTAL 

231 

46 

34 

29 

340 

The  primary  cognitive  activity  is  sensemaking  and  situation  assessment,  where  HSI 
practitioners  perceive,  evaluate,  diagnose,  anticipate,  or  create  data  and  information  in  support  of 
a  tradeoff  decision.  Figure  4  depicts  the  frequency  counts  for  the  identified  sensemaking  themes. 
Note  that  CTR  stands  for  “Contractor”  and  “Govt”  means  “Government. 


Sensemaking  and  Situation  Assessment 


1 

1 

1  1 

n  ,  , 

1  1  1  n  1 

Riskorrisk  Technical  data  Leveraging  Technical  scopeimpact  to  cost,  Design  options  Requirements 
mitigation  previous  of  work,  level  schedule, 

strategy  experience,  of  effort  for  performance 

lessons  learned  CTR,  Govt 


Figure  4:  Sensemaking  and  Situation  Assessment  Themes  and  Frequency  Counts 


It  comes  as  no  surprise  that  evaluating  technical  data  is  the  most  significant  (139 
instances).  Table  4  describes  the  technical  data  theme  in  further  detail.  The  theme  with  the 
second  highest  frequency  count  in  Figure  4  is  assessing  the  impact  of  design  features  on  cost, 
schedule  and  performance.  For  example,  SMEs  reported  (a)  evaluating  the  reliability  of 
technology  and  the  resulting  impact  on  human  performance  effectiveness,  (b)  evaluating 
redesign  efforts  for  requirements  compliance,  (c)  evaluating  and  anticipating  the  impact  of 
having  limited  time,  money  and  people  to  perform  analyses,  and  (d)  evaluating  sensitivities 
around  adding  manpower  and  the  impact  on  total  ownership  costs  and  affordability.  The 
majority  of  these  impact  assessments  were  done  collaboratively  with  other  IPT  members. 
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Table  4:  Subcategories  for  the  Technical  Data  Theme 


Technical  Data  Theme 
Subcategory 

Examples 

HFEs  Primary  (54  instances) 

•  Evaluated  historical  safety  events,  standards,  instructions, 
requirements 

•  Evaluated  workload  reduction  through  usability  testing 

•  Evaluated  technology  readiness  levels  and  levels  of  automation 
(function  allocation) 

•  Perceived/Evaluated  crew  tasks  and  functions 

•  Perceived/Evaluated  environmental  stressors  to  include  in  workload 
analyses 

•  Perceived/Evaluated  complaints  about  the  user  interface 

•  Anticipated/Diagnosed  potential  user  difficulty  with  the  design 

•  Evaluated/Mentally  simulated  interface  use  to  create  a  storyboard 

HFEs  with  Contractor,  IPT, 
Program  Office, 

Management,  other 
Stakeholders  (26  instances) 

•  FIFEs  and  Program  Office  evaluated  risk;  collaborative  discussions  to 
finalize  the  wording  of  the  risk 

•  Perceived/Evaluated  the  interface  design  during  the  Fluman  Machine 
Interface  (FIMl)  working  group  meetings 

•  Evaluated  design  during  Air  Crew  Systems  Advisory  Panel  (ASCAP) 
meetings 

•  Perceived/Evaluated  requirement;  Discussed  requirement  with 
technical  team  (IPT  level) 

•  Perceived/Evaluated  task  analysis  process  during  HFE  group 
brainstorming  session 

Analysis  results 
(28  instances) 

•  Perceived/Evaluated/Questioned  manpower  estimate  created  before 
the  Systems  Requirements  Review  (SRR) 

•  Diagnosed  that  the  baseline  system  used  for  the  initial  manpower 
estimate  had  very  different  attributes  than  current  proposed  system 

•  Evaluated  contractor  equipment  proposal  and  their  justification 

•  Evaluated  data  to  validate  what  was  being  done  during  the  analysis 

•  Evaluated  contractor  response  to  potential  design  change 

•  Anticipated  need  to  instantiate/realize  the  assumptions  made  in  the 
analysis  to  get  desired  workload 

Data  from  Fleet,  User 
Community,  SMEs 
(20  instances) 

•  Perceived/Solicited  design  inputs  through  fleet  user  groups 

•  Perceived/Evaluated  responses  from  users;  Diagnosed  problem  data 

•  SMEs  and  Fleet  representatives  evaluated  prototype 

•  Fleet  perceived/evaluated/raised  concerns  about  the  original  design  to 
the  contractor 

Data  from  literature  review, 
prior  studies,  prior  test 
events  (11  instances) 

•  Perceived/Evaluated  design  using  requirements  and  performance  data 
from  related  research 

•  Perceived/Obtained  data  for  analysis  from  mission  documents, 
tactical  manuals 

•  Diagnosed/Evaluated/Presented  why  the  original  design  was 
insufficient  using  research  and  lessons  learned 

•  Perceived/Evaluated  task  lists  and  the  Fluman  Engineering  Design 
Approach  Document  -  Operator  (HEDAD-0)  for  another  aircraft  to 
support  interface  design 
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Figure  5  shows  that  HSI  practitioners  are  doing  process  planning  as  well  as  HW/SW 
evaluations;  time  is  taken  to  evaluate  the  feasibility  of  courses  of  action  during  the  tradeoff 
process.  This  includes  deciding  on  the  best  path  forward  to  conduct  analyses  and  implement 
design  changes.  For  example,  SMEs  reported  planning  to  conduct  future  modeling  and 
simulation  analysis  and  test  events  to  better  understand  the  full  impact  of  calculated  workload. 
Project  plans  and  schedules  were  also  adapted  because  the  HSI  practitioners/IPT  felt  it  was 
necessary  to  ask  the  contractor  to  verify  the  proposed  changes  before  implementing  them. 


Planning,  Adapting  and  Replanning 


1 

1  1 

1  1 

L._  _  J 

Collaboration/Coordination  with  Process  Used/Path  Forward  Technical  Scope  of  the  Design 

CTR,  IPT,  Management,  other 
Stakeholders 


Figure  5:  Planning,  Adapting  and  Replanning  Themes  and  Frequency  Counts 


The  Uncertainty  and  Risk  Management  themes  correspond  with  the  judgments  identified 
in  the  decision  tables.  As  shown  in  Figure  6,  HSI  practitioners  question  the  accuracy  and 
reliability  of  data  (e.g.  time  on  task  estimates,  assumptions  made),  data  analysis  methods  used 
(e.g.  reproducible,  valid,  reliable),  and  data  analysis  results  (e.g.  number  of  proposed  equipment, 
contractor  provided  cost  estimates).  Technology  maturity,  contractor  reliability  and  the  lack  of 
adequate  baseline  comparison  systems  also  contribute  to  the  uncertainty.  Judgments  were  made 
by  both  HSI  practitioners  and  by  the  greater  IPT  on  whether  or  not  formal  program  risks  needed 
to  be  created,  and  how  best  to  proceed  with  the  tradeoff  in  light  of  the  uncertainty. 

Figure  7  shows  the  themes  for  using  opportunities  and  leverage  points.  HSI  practitioners 
are  actively  coordinating  with  various  program  stakeholders  to  resolve  conflicts  and  achieve  a 
win-win  situation  for  the  tradeoff  In  a  HW/SW  example,  a  prototyping  event  was  conducted  at 
the  same  time  as  a  systems  engineering  technical  review  (SETR)  event.  Because  of  this  timing, 
the  SETR  event  would  not  be  closed  until  the  issues,  major  concerns  and  action  items  related  to 
the  prototyping  event  were  resolved  and  closed.  In  a  manpower/  workload  example,  task 
analyses  were  performed  with  members  of  the  Training  organization  in  order  to  support  the 
development  of  training  master  task  lists. 
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Uncertainty  and  Risk  Management 
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Analysis  method  or  Accuracy  of  data  Proposed  system  Process  to  The  scope  of 

results  used  for  analysis  attributes  Use/Path  Forward  technical/program 
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Figure  6:  Uncertainty  and  Risk  Management  Themes  and  Frequency  Counts 


Using  Opportunities  and  Leverage  Points 

14 
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Coordinating/  Collaborating  Requesting/  Generating/  Requesting/  Generating/ 
on  Design  Options  Leveraging  data  and  data  Leveraging  Design  Options 

results 


Figure  7:  Using  Opportunities  and  Leverage  Points  Themes  and  Frequency  Counts 
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3.4  Relevant  Knowledge  and  Experience 

In  summary,  HSI  practitioners  are  performing  technical,  process,  risk  and  impact 
evaluations  during  tradeoff  decisions.  They  are  determining  when  and  how  to  leverage  previous 
work.  They  are  planning  and  replanning  analysis,  testing  and  implementation  processes.  They 
take  the  time  to  evaluate  current  circumstances  against  previously  experienced  analogous 
situations.  They  also  interface  with  IPT  members,  stakeholders  in  other  organizations,  and 
individuals  they  worked  with  in  the  past  to  gather  and  process  information  and  data.  But,  what 
knowledge,  skills  and  abilities  are  required  to  successfully  execute  these  kinds  of  tasks? 


3.4.1  Experience  and  What-Ifs 

During  the  CDM  interviews,  SMEs  were  asked  to  describe  the  specific  training  or 
experience  they  felt  was  needed  or  helpful  to  them  during  the  tradeoff  incident  they  reported. 
They  were  also  asked  what  they  would  have  done  differently,  what  would  help  another  person 
make  the  same  decision  successfully,  and  what  conditions,  training,  knowledge,  information  or 
tools  might  have  helped  made  the  decision  process  better.  Table  5  lists  the  provided  responses  to 
these  questions.  These  are  not  Knowledge,  Skills  and  Abilities  or  Aptitudes  (KSAs)  as  formally 
defined  by  the  U.S.  Office  of  Personnel  Management  (n.d.)  or  MIL-HDBK  29612  (2001). 
Further  research  would  be  required  to  develop  a  verified,  validated  and  comprehensive  list  of 
HSI  Tradeoff  KSAs. 


Table  5:  Reported  HSI  Tradeoff  Knowledge,  Experience,  Capabilities  and  Tasks 


Knowledge  of... 

Experience  in... 

Additional  Capabilities 
and  Tasks 

•  Workload  modeling 
tools  and  techniques 

•  Where  to  find 
information 

•  Technology  readiness 

•  Design 

•  Human  factors, 
psychology, 
manpower,  personnel, 
training 

•  Human  performance 
measurement,  system 
design  and  training 
development 

•  Workload  analysis 

•  Task  analysis 

•  Doing  literature 
reviews 

•  Understanding 
acquisition  processes 

•  Conducting  research 

•  Tradeoffs 

•  Operational 
experience 

•  The  problem  domain 

•  Being  assertive 

•  Being  patient 

•  Managing  teams 

•  Develop  a  valid  baseline  comparison 

•  Derive  requirements  from  user  inputs 

•  Use  ACSAP  process 

•  Find  Military  Standards  (MIL-STDs) 

•  Investigate  how  manpower  estimate  was 
initially  generated 

•  Have  a  prototyping  capability,  ability  to 
demonstrate  high  fidelity  options 

•  Develop  arguments  to  present  to  program 
office;  Have  a  way  to  determine  cost, 
schedule  and  performance  savings  to 
present  to  the  program  manager 

•  Gather  more  evidence  on  cost  and 
performance;  Investigate  how  the  cost 
estimate  was  generated 

•  Know  how  to  determine  when  the 
contractor  is  right  or  wrong 

•  Choose  words  carefully 

•  Don’t  take  things  personally 

•  Understand  how  to  approach  problems 

•  Know  what  to  focus  on 

•  Be  paired  with  someone  experienced 
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3.4.2  Summarized  Novice/Expert  Differences 

During  the  decision  table  interviews,  SMEs  were  asked  which  actions  described  in  the 
table  clearly  distinguished  experts  from  novices,  and  what  they  thought  novices  might  do  in 
listed  the  activities.  Appendix  E  contains  a  summary  of  the  reported  novice  behaviors  and 
training  needs.  There  was  considerable  overlap  with  the  skills  described  in  the  CDM  interviews. 
However,  the  following  are  additional  knowledge,  experience  and  capabilities  SMEs  stated  as 
required  by  novices  to  successfully  work  on  HSI  Tradeoff  Decisions: 

•  Policy  Familiarity 

•  Selecting  the  best  analysis  method  for  the  situation 

•  Rescoping  tasking  (not  just  rescheduling  tasking) 

•  Assessing  the  impact  of  data  and  data  analysis  on  other  functions 

•  (e.g.  other  HSI  Domains,  Systems  Engineering,  the  contract,  etc) 

•  Judging: 

o  Accuracy  of  data,  data  results,  or  analysis  progress 
o  Adequacy/Accuracy  of  process/analysis  outcomes 
o  Adequacy/Accuracy  of  design  options 

•  Predicting 

•  Process/Analysis  outcomes 

•  Potential  problems 

•  See  the  “big  picture” 

•  Coordinate/Collaborate  with  stakeholders 


4.0  CONCLUSIONS 

This  study  represents  a  significant  first  step  in  trying  to  define  how  HSI  tradeoff 
decisions  are  made.  Due  to  the  number  of  study  participants,  these  results  have  limited 
generalizability,  and  therefore  do  not  characterize  all  possible  HSI  tradeoff  scenarios  within 
Naval  aviation. 

However,  it  is  clear  from  both  the  content  analysis  and  the  decision  requirements  tables 
that  skills  such  as  project  management,  team  management,  communication,  negotiation,  and  risk 
assessment  are  required  to  complement  the  technical  skills  needed  to  perform  HSI  domain  and 
cross-domain  tradeoff  analyses.  It  is  also  apparent  from  the  research  results  that  judgments  of 
data  quality  and  reliability  are  critical  to  the  tradeoff  process;  time  is  also  taken  to  evaluate  the 
impact  of  questionable  data  on  the  design  process.  Considerable  effort  is  also  focused  on 
presenting  cost,  schedule  and  performance  decision  rationales  to  program  management, 
particularly  for  HW/SW  tradeoffs.  Successfully  generating  this  argument  requires  time  and 
resources  to  gather  and  evaluate  information  on  contractor  work  plans,  available  funding,  cost 
estimates  (expenditures  and  savings)  and  quantitative  and  qualitative  HW/SW  and  personnel 
performance  impacts. 

Therefore,  it  would  be  beneficial  to  develop  and  support  these  skills  and  capabilities 
within  the  HSI  workforce.  Further  research  is  needed  to  identify  specific  HSI  tradeoff  KSAs, 
and  determine  which  KSAs  may  require  additional  workforce  training.  For  example,  according 
to  Cannon-Bowers  and  Bell  (1996),  naturalistic  decision  makers  should  receive  training  in  areas 
such  as  metacognitive  skills,  reasoning,  domain-specific  problem  solving,  mental  simulation, 
situation  assessment  and  knowledge  organization.  The  appropriate  types  of  training  would  also 


16 


have  to  be  evaluated.  Means,  Salas,  Crandall  and  Jaeobs  (1993)  reeommend  providing  multiple 
trials  in  problem  reeognition  and  representation  to  develop  skills  for  handling  ill-struetured 
problems,  and  training  for  monitoring,  eommunieating,  and  providing/reeeiving  feedbaek  to 
handle  multiple  players  involved  in  the  deeision  making  proeess.  Finally,  the  feasibility  of 
providing  this  additional  training  would  have  to  be  evaluated.  Within  any  organization,  it  is  a 
Management  and  Human  Resourees  deeision  to  either  hire  individuals  with  demonstrated 
performanee  in  sueh  skills,  or  to  provide  profieieney  training  for  new  and  existing  employees. 

Additional  researeh  and  development  in  HSI  speeifie  eost  and  risk  assessment  deeision 
support  tools  may  also  benefit  HSI  praetitioners.  For  example,  a  human  performanee  assessment 
methodology  or  tool  that  quantifies  and  qualifies  eost  and  risk  at  different  points  in  the 
aequisition  program  using  faetors  sueh  as  teehnology  maturity,  data  validity,  and  pereeived 
reliability  of  assumptions  would  allow  HSI  praetitioners  to  present  more  reliable  assessments  to 
the  Program  Offiee.  Collaborative  partnerships  with  universities  and  other  organizations 
eurrently  working  in  this  area  would  further  support  future  researeh  in  HSI  Tradeoffs. 
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APPENDIX  A  -  Critical  Decision  Method  and  Decision  Table  Interview 
Questions 


Demographic  Questionnaire 

Participant  # _ 

1 .  What  is  your  educational  background  (e.g.  B.S.  American  Studies,  M.S.  Industrial 
Engineering)? 

2.  How  many  years  of  DoD  acquisition  experience  do  you  have? 

3.  List  the  different  roles/positions  you  had  during  your  years  of  acquisition  service 

4.  How  many  years  of  specifically  Naval  Aviation  work  experience  do  you  have? 

5.  How  many  years  of  specifically  HSI  experience  do  you  have? 


Interview  and  Critical  Decision  Method  Prohe  questions 

1.  Incident  Selection:  The  interviewer  will  ask  the  SME  to  select  a  HSI  tradeoff  that  occurred 
within  the  last  5  years.  The  SME  will  be  asked  to  answer  the  following  questions: 

a.  What  was  the  Naval  Aviation  acquisition  and  what  type  of  program  was  it  (e.g. 
ACAT  I,  II,  etc)? 

b.  Did  the  tradeoff  decision  concern  a  component,  subsystem,  system,  family  of 
systems,  system  of  systems,  process  or  procedure? 

c.  Which  HSI  domains  were  involved  in  the  tradeoff  decision? 

d.  What  was  your  role  on  the  program  when  the  decision  occurred? 

e.  How  many  years  of  experience  did  you  have  on  that  specific  acquisition  program 
when  the  decision  occurred?  How  many  years  of  total  acquisition  experience  did  you 
have  when  the  decision  occurred? 

2.  Incident  Recall:  The  SME  will  be  asked  to  describe  the  events  that  occurred  before,  during 
and  immediately  after  the  decision  was  made. 

3.  Clarification:  The  interviewer  will  be  asked  to  retell  the  incident,  focusing  on  the  timeline, 
critical  incidents  and  decision  points.  The  SME  will  have  the  opportunity  to  clarify,  correct 
or  add  more  information. 

4.  Probe  Questions:  The  interviewer  will  ask  the  probe  questions,  allowing  the  SME  to  further 
elaborate  on  the  critical  incidents,  decision  points  and  other  factors  that  affected  the  decision 
making  process.  The  probe  questions  were  adapted  from  Hoffman,  et  al  (1998),  O’Hare  et  al, 
(1998)  and  Taylor  (2007).  Questions  for  clarification  (e.g.  How?  Why?)  will  be  asked  when 
appropriate. 

5.  What  if  queries:  The  interviewer  will  ask  the  hypothetical  questions,  allowing  the  SME  to 
speculate  on  how  their  decision  making  process  could  have  been  better  or  how  the  decision 
may  have  been  different  if  key  features  of  the  situation  were  different. 
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Probe  Type 

Probe  Content 

Personnel  Interactions 

Did  you  have  the  final  say  in  the  decision?  If  no,  which  other  personnel  were 
the  key  players  in  the  decision? 

Planning 

Had  you  or  anyone  else  anticipated  the  possibility  of  this  tradeoff  earlier  in 
the  acquisition  timeline?  Were  any  contingency  plans  made?  What  happened 
to  those  plans? 

Situation  cues 

What  factors  facilitated  the  need  for  the  tradeoff? 

How  did  you  know  for  sure  that  a  tradeoff  was  necessary? 

Goals 

What  were  the  specific  goals  and  objectives  when  the  decision  making 
process  began? 

Analogs 

Did  this  situation  remind  of  any  previous  work  experience?  If  yes,  how  did  it 
impact  your  approach  for  this  decision? 

Information 

How  did  you  determine  what  information  was  required  to  make  the  decision? 
How  was  the  information  obtained? 

Influence  of  uncertainty 

At  any  stage,  were  you  uncertain  about  either  the  reliability  or  the  relevance 
of  information  that  you  had  available?  If  yes,  what  happened? 

Options 

What  limitations  did  you  face  regarding  possible  alternatives?  What  other 
courses  of  action  were  considered  or  were  available? 

Decision  making 

Were  there  any  factors  that  impacted  the  decision  making  process? 

How  much  time  pressure  was  involved  in  making  this  decision?  How  long  did 
it  take  to  actually  make  this  decision? 

Errors 

Were  possible  mistakes  in  the  decision  process  anticipated?  How  were  they 
avoided?  Did  any  mistakes  occur?  If  yes,  how  were  they  corrected? 

Basis  of  choice 

How  was  the  final  option  selected  and  other  options  rejected?  What  rule  was 
being  followed?  What  was  your  specific  contribution? 

Mental  modeling 

Did  you  imagine  the  possible  consequences  of  the  final  option?  Did  you 
imagine  the  events  that  would  unfold  as  a  result? 

Results  of  actions 

Did  the  tradeoff  work  as  expected?  If  not,  what  might  have  caused  it  not  to 
work? 

Experience 

What  specific  training  or  experience  was  necessary  or  helpful  in  making  this 
decision? 

Do  you  think  that  you  could  develop  a  rule,  based  on  this  experience,  which 
could  assist  another  person  to  make  the  same  decision  successfully?  Do  you 
think  that  anyone  else  would  be  able  to  use  this  rule  successfully?  Why?  Why 
not? 

Hypotheticals 

With  hind-sight,  what  would  you  have  done  differently? 

What  conditions,  training,  knowledge,  information  or  tools  might  have  helped 
made  the  decision  process  better? 

What  would  have  happened  if  the  tradeoff  hadn’t  worked?  What  would  you 
have  done? 
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Decision  Table  Verification  and  Validation  Interview  Questions 

After  the  informed  eonsent  forms  have  been  signed,  the  interviewer  will  explain  to  the  SME  the 
baekground  of  the  ILIR  and  the  eurrent  progress  to  date.  The  interviewer  will  then  review  the 
deeision  tables  with  SME. 

For  eaeh  aetivity  listed  in  the  table,  the  interviewer  and  SME  will  diseuss  the  eues,  judgments, 
ehallenges  and  strategies  used  to  address  the  ehallenges  that  were  diseovered  using  the  five  ease 
studies.  Diseussion  will  eenter  around  the  following  questions: 

•  Do  you  have  any  eomments  about  the  eues,  judgments,  ehallenges  and  strategies  listed  in 
the  table?  Is  there  anything  you  think  should  be  added  or  removed?  Please  provide 
reasons  why. 

•  Are  there  any  other  aetivities  that  you  think  should  be  added  to  this  table?  Why? 

•  What  faetors  do  you  think  impaet  the  various  aetivities  listed  in  the  table?  Why? 

•  Whieh  aetions  deseribed  in  this  table  do  you  think  are  the  most  ehallenging?  Why? 

•  Whieh  aetions  deseribed  in  this  table  do  you  think  elearly  distinguish  experts  from 
noviees?  Why? 

•  Considering  the  list  of  experienee  used  and  useful  tools/teehniques/skills,  is  there 
anything  you  would  add  to  this  list?  Why? 
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APPENDIX  B  -  Summarized  Responses  to  the  CDM  Interview  Probe  Questions 


Question 

Workload  Analyses 

HW/SW/Process  Changes 

Summary  of  T radeoff  Decision 

These  were  not  tradeoff  decisions  per  se.  They 
were  verifications  that  the  planned  crew  size  was 
sufficient  to  perform  the  mission  and  that  the 
workload  was  reasonable  considering  the 
mission  and  system  design. 

HW/SW/Process/Procedure  design  changes  for 
requirements/standards  compliance,  workload 
mitigation,  improved  usability,  and/or  to 
minimize  impact  to  training  design 

Interact  1:  Did  you  have  the  final  say  in 
the  decision? 

Yes/  Will  have  a  final  say  for  analyses  not 
complete  at  the  time  of  the  study 

Yes/  Will  have  a  final  say  for  analyses  not 
complete  at  the  time  of  the  study 

Interact  la:  If  no,  which  other  personnel 
were  the  key  players  in  the  decision? 

Even  though  participants  had  or  will  have  a  final 
say,  other  personnel  were/will  always  part  of  the 
decision: 

Even  though  participants  had  or  will  have  a  final 
say,  other  personnel  were  always  part  of  the 
decision: 

Human  Factors  Engineers  and  their  management 
(Gov't  and  Industry) 

Human  Factors  Engineers  and  their  management 
(Gov't  and  Industry) 

Fleet  Representatives 

Fleet  Representatives 

Subject  matter  experts  (Gov't  and  Industry) 

Subject  matter  experts  (Gov't  and  Industry) 

IPT  members  from  other  NAVAIR  departments, 
NAVY  program  offices  and 
Contractor/Subcontractor  organizations 

IPT  members  from  other  NAVAIR  departments, 
NAVY  program  offices  and 
Contractor/Subcontractor  organizations 

Plan  2:  Had  you  or  anyone  else 
anticipated  the  possibility  of  this 
tradeoff  earlier  in  the  acquisition 
timeline? 

Evidence  existed  but  formal  analysis  was  not 
requested  until  a  program  office  risk  was  written 

No 

Evidence  that  Fleet  representatives  had 
suspicions. 

Plan  3:  Were  any  contingency  plans 
made? 

No 

No 

Plan  3a:  What  happened  to  those  plans? 

N/A 

N/A 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

Cue  1:  What  factors  facilitated  the  need 
for  the  tradeoff?  [What  was  seen  or 
heard?] 

Comparison  made  to  similar  systems 

Information  from  the  research  literature 

Actual  research  and  test  results 

Fleet  inputs 

Formal  risk  identified  and  documented 

Judgments  from  lessons  learned  and  experience 

Current  design  reviewed 

Cue  2:  How  did  you  know  for  sure  that  a 
tradeoff  was  necessary? 

The  program  could  not  move  forward  without 
verified  and  validated  manpower  and  workload 
estimates 

Design  review  highlighted  non  compliance  with 
requirements  and  standards 

Personal  and/or  team  judgment  of  potential 
consequences  of  current  design 

Goal  1:  What  were  the  specific  goals  and 
objectives  when  the  decision  making 
process  began? 

Quantify,  verify,  validate  the  workload  and 
manpower  estimates 

Verify  requirements 

Generate/Propose  design  options  to  meet 
requirements  and  standards,  to  improve 
usability,  to  manage  user  workload 

Solicit  Fleet  and  SME  feedback  on  design 
options 

Analog  1:  Did  this  situation  remind  of 
any  previous  work  experience? 

Yes 

Yes 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

Analog  la:  If  yes,  how  did  it  impact  your 
approach  for  this  decision? 

Lessons  learned  from  previous  experience 
impacted  the  level  of  detail  and  thoroughness  of 
the  analysis 

Previous  experience  impact  the  data  collection 
approach  -  made  sure  to  review  additional 
standards,  requirements,  technology  readiness 
levels,  previous  research  reports,  literature 
review 

Lessons  learned  from  previous  experience 
placed  an  emphasis  on  getting  Fleet  and 
management  support  on  the  issue  and  design 
options  prior  to  presentation  to  the  program 
office 

Info  1:  How  did  you  determine  what 
information  was  required  to  make  the 
decision? 

Lessons  learned  from  previous  experience  in 
workload  and  manpower  analyses  was  used  to 
determine  what  information  was  required 

Consideration  given  on  what  would  be  needed  to 
create  a  usable  system 

Consideration  given  on  what  would  be  needed  to 
create  a  comprehensive  argument  to  the 
contractor  and  program  office 

Information  needed 

Use  cases/Scenarios/Task  Descriptions 

Requirements  and  standards 

System  characteristics  (actual,  projected) 

Safety  Data/Hazard  Risk  Index 

Who  does  what  (human  vs.  system)? 

Task  Descriptions 

Best  approximations/estimates  of  time  on  task 
under  different  environmental  conditions 

System  characteristics  (actual,  projected) 

Design  characteristics  of  similar  systems 

Previous  test  data  and  research  results 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

Info  la:  How  was  the  information 

Obtained/Requested  from: 

Obtained/Requested  from: 

obtained? 

•  Mission  description  documents 

•  Mission  description  documents 

•  User's  Tactical  Manuals 

•  Human  Engineering  Design  Approach 

•  SMEs 

Document  -  Operator  (HEDAD-0) 

•  Fleet  Representatives 

•  User's  Tactical  Manuals 

•  Contractors 

•  SMEs 

•  Other  IPT  members 

•  Fleet  Representatives 

•  Contractors 

•  Other  IPT  members 

Analyses  Performed 

Baseline  comparison  analysis 

Reviewed  current  design 

Timeline  analysis 

Compared  and  contrasted  different  technologies 

Mission  decomposition  analysis/Crew  task 

Conducted  a  prototype/storyboard  review 

analysis 

Conducted  a  technology  demonstration 

Function  allocation  analysis 

Generated  design  options 

User  interface/Decision  aid  evaluations 

Group  consensus  on  best  option 

Created  presentation  to  the  program  office 

Created  white  paper/presentation  for  the 
program  office  describing  the  pros  and  cons  of 
the  options  and  why  one  is  best. 

Uncertainl :  At  any  stage,  were  you 

Uncertain  about  contractor  estimates  for  timeline 

Uncertain  about  the  contractor  information  on 

uncertain  about  either  the  reliability  or 
the  relevance  of  information  that  you 

analysis 

technology  use 

had  available? 

Uncertain  about  projected  HW/SW  capabilities 

Uncertain  about  contractor  estimate  of  level  of 
effort 

Uncertain  about  the  existence  of  unique  system 
requirements 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

Uncertainla:  If  yes,  what  happened? 

Contractor  questioned,  but  had  to  accept  the 
estimates 

Had  to  accept  at  face  value  and  use  the  data 

Contractor  questioned,  but  had  to  accept  the 
estimates 

Consulted  with  other  IPT  members  about 
requirements 

Option  1 :  What  limitations  did  you  face 
regarding  possible  alternatives? 

Additional  cost  for  out  of  scope  contractor 
tasking 

Parts  of  the  system  could  not  be  modified 

Some  design  options  were  infeasible  due  to 
impact  on  interfaces  with  other  systems 

Some  changes  to  operational  process/procedure 
were  infeasible  due  to  downstream  impacts 

Option  2:  What  other  courses  of  action 
were  considered  or  were  available? 

Workload  analysis  had  to  be  done  -  no  option 
there 

Considered  multiple  crew  configurations  but  no 
separate  comparative  analyses  completed 

Doing  nothing  was  determined  to  be  an 
infeasible  option 

Redesigning  the  system/component  and  creating 
multiple  redesign  options  was  the  only  option 

Decision  1:  Were  there  any  factors  that 
impacted  the  decision  making  process? 

Limited  funding  and  resource  availability  (Gov't 
and  contractor) 

Fleet  input  needed  for  verification 

Sensitivity  around  the  cost  of  adding  manpower 
to  the  program 

Tension  around  impending  contractual  changes 
and  the  resulting  cost  implications  in  order  to 
implement  system  redesign 

Anticipating  contractor  and/or  program  office 
pushback  and  preparing  adequate  counter 
arguments 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

Decision  2:  How  much  time  pressure 
was  involved  in  making  this  decision? 

Time  pressure  to  review  contractor 
documentation  in  accordance  with  review 
schedule 

Need  to  complete  analyses  to  support  next 
milestone  decision 

Need  revised  manpower  estimate  (based  on 
workload  analysis)  since  already  post  contract 
award 

Time  pressure  to  complete  work  within  schedule 

Need  to  complete  analysis  before  next  technical 
review  and/or  test  event 

No  time  pressure 

Decision  3:  How  long  did  it  take  to 
actually  make  this  decision? 

Analysis  not  completed  at  time  of  study 

Analysis  completed  within  one  year 

Analysis  completed  in  six  months 

Analysis  completed  in  one  month 

Analysis  completed  in  three  months 

Error  1:  Were  possible  mistakes  in  the 
decision  process  anticipated? 

Anticipated  that  the  desired  data  would  not  be 
obtained  due  to  lack  of  funding  and  resource 
availability 

No 

Error  2:  How  were  they  avoided? 

Not  avoided 

N/A 

Error  3:  Did  any  mistakes  occur? 

No 

Not  finding  design  problem  sooner 

Not  understanding  the  contractor's  work 
breakdown  structure  (WBS)  and 
development/delivery  schedule 

Error  3a:  If  yes,  how  were  they 
corrected? 

N/A 

As  soon  as  problem  was  discovered,  redesign 
became  a  priority  task 

Contractor  schedule  and  WBS  information 
became  available  indirectly  through  a  request  for 
action 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

Choice  1:  How  was  the  final  option 
selected  and  other  options  rejected? 

N/A 

Team  consensus 

Cost  evaluation 

Choice  2:  What  rule  was  being  followed? 

N/A 

Ensured  design  was  requirement,  standards,  and 
safety  compliant 

Ensured  design  aligned  with  Fleet  feedback 

Choice  3:  What  was  your  specific 

Identified  the  consequences  of  not  mitigating  the 

Human  factors  engineering  evaluations 

contribution? 

workload  risk 

Key  driver/facilitator  for  design  option 

VerifiedA^alidated  the  analysis  results  and 
highlighted  the  implications  of  the  analysis 
results  for  the  program 

generation 

Model  1 :  Did  you  imagine  the  possible 

Identified  the  consequences  of  not  mitigating  the 

Considered  design  implications  on  user  safety 

consequences  of  the  final  option? 

workload  risk 

Used  a  database  tool  to  characterize  and  model 
workload  and  perform  sensitivity  analyses 

and  user  performance 

Model  2:  Did  you  imagine  the  events 

Used  feedback  from  Fleet  representatives  and 

Envisioned  how  user  would  interact  with  each 

that  would  unfold  as  a  result? 

users  of  similar  systems;  also  used  information 
for  relevant  literature  and  lessons  learned 

Performed  sensitivity  (what  it)  analyses 

design  option 

Result  1 :  Did  the  tradeoff  work  as 
expected? 

N/A 

N/A 

Yes  -  Crew  size  was  determined  to  be  adequate 
for  the  system  design  with  predicted  capabilities 

Yes 

Yes 

Result  la:  If  not,  what  might  have 
caused  it  not  to  work? 

N/A 

N/A 
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Question 

Workload  Analyses  HW/SW/Process  Changes 

Experience  1:  What  specific  training  or 

Experience  in: 

experience  was  necessary  or  helpful  in 

• 

Workload  and  task  analysis 

making  this  decision? 

• 

Conducting  research  and  literature  reviews 

• 

Human  factors,  psychology,  manpower,  personnel,  training.  Navy  operations 

• 

Conducting  tradeoff  analyses 

• 

Understanding  acquisition  processes 

• 

Being  assertive 

• 

Being  patient 

Knowledge  of: 

• 

Workload  modeling  tools  and  techniques 

• 

Where  to  find  information 

• 

Technology  readiness 

• 

System  design 

Experience  2:  Do  you  think  that  you 

• 

Do  task  analysis 

could  develop  a  rule,  based  on  this 

• 

Develop  a  valid  baseline  comparison 

experience,  which  could  assist  another 

• 

Derive  requirements  from  user  inputs 

person  to  make  the  same  decision 

• 

Use  Air  Crew  Systems  Advisory  Panel  (ACSAP)  process 

successfully? 

• 

Find  Mil-Standards 

• 

Develop  arguments  to  present  to  program  office 

• 

Know  how  to  determine  when  the  contractor  is  right  or  wrong 

• 

Choose  words  carefully 

• 

Don’t  take  things  personally 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

Experience  3:  Do  you  think  that  anyone 
else  would  be  able  to  use  this  rule 
successfully?  Experience  3a:  Why?  Why 
not? 

Need  experience  in: 

•  Managing  teams 

•  Human  performance  measurement 

•  System  design  and  training  development 

Need  relevant  education,  experience  and  exposure 

Need  to: 

•  Understand  how  to  approach  problems 

•  Know  what  to  focus  on 

•  Be  paired  with  someone  experienced 

What  If  1:  With  hind-sight,  what  would 
you  have  done  differently? 

Gathered  more  evidence  on  cost  and  performance  of  other  design  options 

Investigated  cost  data  and  how  the  cost  estimate  for  design  options  were  generated 

Found  the  problem  sooner,  particularly  when  examining  analogous  situation 

Investigated  how  manpower  estimate  was  initially  generated 

Done  more  homework  on  requirements  and  contractor  work  plans  and  funding  levels 

What  If  2:  What  conditions,  training, 
knowledge,  information  or  tools  might 
have  helped  made  the  decision  process 
better? 

Having  a  way  to  determine  cost,  schedule  and  performance  savings  to  present  to  the  program 
manager 

Having  training  in  human  factors 

Having  experience  in  the  problem  domain 

Having  funding  and  resources 

Having  a  prototyping  capability;  Demonstrating  high  fidelity  options;  Doing  a  prototype  assessment 
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Question 

Workload  Analyses 

HW/SW/Process  Changes 

What  If  3:  What  would  have  happened  if 
the  tradeoff  hadn’t  worked? 

What  If  3a:  What  would  you  have  done? 

Program  Risk  would  have  remained  until  another  mitigation  found 

Workload  risk  would  have  been  created 

System  may  have  not  passed  the  next  technical  review  or  failed  developmental  testing 
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APPENDIX  C  -  Decision  Requirements  Tables 


Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties 
Strategies/Mitigations 

Workload/Manpower: 

Examine/  Investigate 
initial  manpower 
estimate  and  the 
assumed  workload  for 
the  intended  system 
design 

A  policy  or  statue 
requiring  a 
manpower  analysis 

A  program 
manpower  Key 
Performance 
Parameter  (KPP) 
(either  exists  or  not) 

A  program  level 
manpower  risk 
(either  exists  or  not) 

The  Analysis  of 
Alternatives  (AoA) 
results  favor  a 
system  design  that 
requires  a  reduced 
crew  size. 

Stakeholders 

(NAVAIR 

manpower, 

NAVMAC,  HFEs, 
SMEs,  Fleet  Reps, 
etc)  judge  the 
manpower  estimate 
to  be  invalid. 

A  lack  of  supporting 
analysis  for  the 
existing  manpower 
requirement  or  the 
manpower  estimate 

The  AoA  and 

Doctrine, 

Organization, 

Training,  Materiel, 
Eeadership, 

Personnel,  Facilities 
(DOTMEPF)  results 

The  characteristics  of: 

•  the  baseline 
comparison  system 
(has  very  different 
attributes  than  the 
proposed  system) 

•  the  billets  under 
consideration 

•  Big  Navy 
personnel 
requirements 

•  operational  tempo 
and  mission  needs 

Using  lessons  learned 

Judge  the  validity  of  the 
process  and  methodology 
used  to  do  the  manpower 
analysis  (not  a  critique  of 
the  end  result  at  this 
point).  Suspicion  in  the 
process  used  would  make 
the  manpower  number  or 
end  result  questionable. 

Judge  the  validity  of  the 
manpower/workload 
assumptions  made 

Judge  the  adequacy  of  a 
demonstrated 
understanding  of  the 
concept  of  operations 
(CONOPS)  and  the 
intended  use  of  the 
system  within  that  context 

Judge  how  the 
platform/system  would 
operate  within  the  battle 
space. 

Judge  the  level  of 
automation  and 

•  There  is  little  confidence  in  the 
resource  information  and  data  used  in 
the  analysis. 

o  Strategy:  Invoke  a  different 
analysis  to  pull  in  credible 
information  and  artifacts. 

•  The  concept  of  operations  is  not  very 
detailed  and/or  largely  fluid  at  any 
point  in  the  analysis. 

o  Strategy:  Characterize  the  analysis 
results  relative  to  what  is  known, 
and  understand  the  limitations  of 
the  analysis  and  the  estimate  itself 
Include  a  range  of  possible 
outcomes. 

•  Stakeholders  do  not  see  the  value  in 
evaluating  or  changing  the  manpower 
levels  and  do  not  support  conducting 
the  analysis. 

o  Strategy:  Estimate  the  impact  of 
performing  a  manpower  analysis. 
Explain  how  a  change  would 
benefit  the  crew  and  the  program. 
Use  a  lot  of  face  time  and  tailor  the 
conversation  depending  on  the 
stakeholder  (e.g.  use  a  scientific 
approach,  personal  approach,  etc). 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties 
Strategies/Mitigations 

A  recognized  need  to 
verify  assumptions 
on  the  number  of 
crew  and  crew 
capabilities 

New  prototype  test 
results 

Changes  in: 

•  program  funding 
(plus-up  or  plus- 
down) 

•  the  system’s 
design 

•  mission 
parameters 

from  previous 
experience  to  guide 
the  analysis 

technology  immaturity  to 
determine  whether  or  not 
the  manpower  will  be 
able  to  support  the  task 
and  missions. 

•  There  is  insufficient  funding  to  do  the 
analysis. 

o  Strategy:  Document  it  as  a  risk; 
petition  for  additional  funding. 

•  The  baseline  system  is  very  different 
than  the  new  proposed  design,  which 
doesn’t  exist. 

o  Strategy:  Conduct  multiple  focus 
groups  with  the  Fleet/SMEs. 

Perform  detailed  task  analyses,  to 
understand  the  task,  the  people  and 
the  mission.  Coordinate  with 
HW/SW  engineers  to  understand 
the  unknowns  and  deltas. 
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Activities 

Workload/Manpower: 

Create  Program  Risk 
for  Manpower 


Cues 

What  triggers  this 
_ activity? 

HFE(s)  and  other 
IPX  members  raise 
concerns  about 
workload  and 
manpower 


Factors 

What  influences  this 
_ activity? _ 

The  existence  of  a 
strong  manpower 
requirement  or  KPP 

The  availability  of 
workload  or  task  data 


Judgments 


Judge  whether  the 
program  office  will 
approve  the  use  of 
resources  to  do  a 
confirmation  study. 


A  systems 

engineering  technical 
review  board 
identifies  a  need  for 
a  manpower  risk  and 
mitigation  strategy 

A  lack  of  supporting 
analysis  for  the 
existing  manpower 
requirement  or  the 
manpower  estimate 

The  results  of  a 
manpower  model, 
simulation  model, 
concept  of 
operations  exercise 
or  other  usability  test 
indicates  the 
potential  for  a  risk 

The  program  does 
not  have  enough 
resources  to 
adequately  assess  the 


Leveraging  data  from 
previous  workload 
studies 

Using  lessons  learned 
from  previous 
experience  to  gauge 
the  likelihood, 
severity  and 
consequences  of  the 
risk 

The  need  to  formally 
track  manpower 
concerns 

The  need  to  generate 
different  manpower 
and  system  design 
strategies  by  the  next 
milestone  review  if 
the  manpower 
assumptions  cannot 
be  verified. 


Judge  whether  or  not 
the  design  supports 
the  current 
manpower. 

Judge  the  likelihood 
of  a  problem  if  the 
workload  is 
determined  to  be 
greater  than  number 
of  crew  allocated 

Judge  the  impact  of 
the  manpower  levels 
on  the  design  itself. 

Judge  the  impact  of  a 
manpower  increase 
or  decrease  on 
warfighters’  quality 
of  life 

Judge  the  potential 
cost  of  additional 
manpower  and 
impact  to  the 


Challenges/  Difficulties 
Strategies/Mitigations 


Neither  the  Government  nor  the  contractor 
is  fiinded  to  perform  the  desired  level  of 
workload  or  manpower  studies, 
o  Strategy:  Contractors  perform  planned 
analyses  within  contract  scope, 
leveraging  previous  studies  and 
literature  reviews. 

Stakeholders  disagree  on  the 
characterization  of  the  risk: 

>  Not  enough  equipment  to 
accommodate  the  missions  versus 
not  enough  people  to  do  the 
mission 

>  Whether  or  not  an  issue  is  actually 
a  risk 

>  Who  should  own  the  risk 

>  Workload  focus:  utilization  versus 
cognitive  workload 

o  Strategy:  Conduct  multiple  discussions 
until  group  consensus  is  achieved. 
Involve  HFE  management  as  needed  to 
facilitate/mediate  the  discussions. 

It  unclear  how  best  to  demonstrate  that 
adding/removing  people  or  changing  the 
design  will  mitigate  the  manpower  risk, 
o  Strategy:  Define  the  options,  work 
through  them  via  analysis,  then  do  a 
real  live  test  of  some  sort,  using  a  score 
sheet  to  evaluate  options. 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties 
Strategies/Mitigations 

task  environment 

The  need  to  tradeoff 
interface  design  and 
mission  capabilities 
(can’t  have  or  do 
ever5^hing  within  the 
task  environment) 

program 

Judge  what 
mitigation  steps  are 
available  and 
whether  or  not  they 
can  get  approved  by 
the  program  office. 

•  The  risk  is  not  accepted  by  the  program 
office  because  it  could  not  be  proven  that 
the  proposed  upgrades  would  support  the 
available  manpower, 
o  Strategy:  Involve  HFE  leadership  in 
discussions  with  Program  Office  to 
facilitate/mediate  discussions. 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties 
Strategies/Mitigations 

Workload/ 

Manpower: 

Perform 
analyses  to 
confirm 
workload 
and 

manpower 
for  the 
intended 
system 
design 

A  program  level 
manpower  risk 

A  requirement 
mandating  a 
manpower  analysis 

The  Analysis  of 
Alternatives  (AoA) 
results  favor  a 
system  design  that 
requires  a  reduced 
crew  size. 

A  recognized  need 
to  validate  the 
current  manpower 
estimate  or  revise 
the  manpower 
estimate 

A  recognized  need 
to  verify 

assumptions  on  the 
number  of  crew  and 
crew  capabilities 

Using  intuition  and 
lessons  learned  from 
previous  experience, 
a  determination  is 
made  that  further 

Available  time,  money 
and  resources  to  do  the 
analysis 

Available  software 
and  analysis  methods 
to  do  the  analysis 

Whether  or  not  the 
analysis  results  will 
impact: 

•  the  work  of  other 
IPT  members 
and/or  program 
stakeholders 

•  rate  retention  for 
relevant  billets 

The  need  to 
communicate  the 
impacts  of  the 
manpower  risk  to 
program  office  and 
other  Navy 
organizations 

The  need  to  generate 
different  manpower 
and  system  design 
strategies  by  the  next 
milestone  review  if  the 
manpower 

Judge  the  quality  and 
reliability  of: 

•  the  method  to 
determine  which 
tasks  to  focus  on 
for  the  analysis, 

•  the  scope  of  the 
analysis, 

•  the  data  used  for 
the  analysis, 

•  the  data  sources 
(include 
personnel 
backgrounds  and 
experience), 

•  the  assumptions, 
predictions  and 
approximations 
made  about  the 
system's 
capabilities, 

•  the  time  on  task 
and  workload 

assessments 

Judge  the  quality  of 
the  workload  and 
manpower  analyses 
considering: 

•  confidence  in  the 
contractor, 

•  what  can  be 

•  HFEs  and  Non-HFE  stakeholders  do  not  agree  on  the 
specific  tasks  to  analyze  (i.e.  can’t  include  every  task) 

o  Strategy:  Discuss,  scope  down,  prioritize,  and  gain 
consensus. 

•  The  task  model  needs  to  be  accurate  and  validated. 

o  Strategy:  Use  reliable  and  established  modeling 

techniques.  For  the  task  analysis,  rely  on  SMEs  for 
task  flow  and  timings.  Validate  against  human  in 
the  loop  testing. 

•  The  contractor  puts  off  doing  work  that’ s  in  the 
contract. 

o  Strategy:  Fight  for  them  to  do  it,  move  the 
deliverable  up  in  the  schedule,  or  work  with 
available  information  (do  without  the  analysis  until 
it  is  completed) 

•  The  contractor’s  timeline  analysis  lacks  rigor  (i.e.  it  is 
quick  and  dirty). 

o  Strategy:  Government  and  contractor  discuss  the 
validity  of  data  sources  and  analyses  used.  Find 
leverage  within  standards  and  requirements  to 
determine  whether  or  not  the  workload  estimates 
are  accurate  or  if  contractual  obligations  have  been 
met.  If  subject  to  cost  and  schedule  constraints, 
accept  the  data  as  is. 

•  SMEs  and  users  provide  questionable  input  for  the 
workload  analysis 

o  Strategy:  Interview  multiple  SMEs/Fleet  reps. 
Remove  inconsistent  data  as  outliers  or 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties 
Strategies/Mitigations 

analysis  should  be 
done. 

assumptions  cannot  be 
verified. 

accomplished 
with  available 
time,  money,  and 

verify/validate  with  other  users 

•  The  analysis  reveals  a  need  to  modify  the  design  in  a 

The  contract 

Using  lessons  learned 

resources 

way  that  is  beyond  the  original  contract  scope. 

requires  the 

from  previous 

•  planned 

o  Strategy:  Make  a  formal  request  to  the  program  to 

completion  of 

experience  to  guide 

modeling  and 

add  scope  to  the  contractor's  statement  of  work 

workload  studies  to 
determine  the  actual 

the  analysis  and  the 
level  of  detail  in  the 

simulation 
efforts  and  the 

(contract  modification) 

number  of 
equipment  needed. 

analysis. 

next  major  test 
event  will 
validate  the 
workload  and 
manpower 

•  There  is  a  need  for  a  revised  manpower  estimate,  even 
though  the  workload  analysis  is  not  yet  completed, 
o  Strategy:  Perform  a  parallel  effort  to  understand  the 
potential  negative  impacts  of  the  type  of  work, 
workload  and  work  circumstances;  revise  the 
estimate. 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties 
Strategies/Mitigations 

HW/SW 

Changes: 

Evaluate 
initial  system 
design  for 
potential 
problems 

A  requirement  to 
review  the  design  as 
part  of  the  normal 
systems  engineering 
process. 

The  phase  of  the 
acquisition  program 

The  contractor 
provides  either  one 
design  or  proposes 
multiple  design 
options  and 
strategies. 

A  capability  need 
from  the  user 
community  doesn’t 
exist  in  the  design  or 
the  requirements,  and 
requires  a  change  to 
the  current  HW/SW 
configuration. 

A  need  to  review 
proposed  designs  as 
part  of  a  source 
selection  process. 

Contractor/sub 
contractor  performance 
and  experience  with 
system 

The  contractor’s 
understanding  of  the 
design  requests,  the  user 
community  and  the 
users’  tasks 

The  availability  of  Fleet 
Reps/SMEs  to  review 
the  design 

The  level  of 

Government  interest  and 
involvement  in  the 
design  process. 

The  visibility  of  the 
system  by  the  program 
office  or  key 
stakeholders. 

The  cost  to  evaluate  the 
system  or  the  make  a 
HW/SW  or  schedule 
change 

A  need  to  control 
requirements  creep. 

Judge  which  approach 
should  be  used  to 
evaluate  the  design 
(focus  group,  prototype 
testing,  simulation,  etc) 

Judge  the  adequacy  of  a 
paper  analysis  to  assess 
the  design  versus  using 
an  actual  prototype 

Judge  which 
stakeholders  need  to 
participate  in  the  design 
evaluation  from  the 
Government, 
contractor.  Fleet,  etc. 

Judge  how  the  design 
compares  to  that  of  a 
similar  system 

Judge  the  quality  of  the 
contractor’ s  design,  and 
their  trustworthiness 
based  on  past 
performance 

•  There  is  not  enough  time  or  money  to 
evaluate  the  design. 

o  Strategy:  Request  more  time/money 

•  There  is  a  lack  of  detail  in  the  design  and  lots 
of  assumptions 

o  Strategy:  Try  to  get  more  details;  Ask  for 
validation  of  assumptions 

•  There  is  no  design  prototype  available. 

o  Strategy:  Use  more  experienced  people  to 
evaluate  the  design. 

•  More  experienced  SMEs  who  can  actually 
mentally  simulate  the  use  of  the  proposed 
system  are  not  available. 

o  Strategy:  HFEs  extract  what  knowledge 
they  can  from  less  experienced 

SMEs/Fleet  Reps,  then  guide  them 
through  a  mental  simulation. 

•  Stakeholders  outside  of  the  program  attempt 
to  influence  the  design  and  the  way  it  is 
evaluated. 

o  Strategy:  Involve  people  who  have  strong 
leadership  and  coordination  skills  and 
very  strong  technical  backgrounds  to 
screen  unreasonable  requests.  Request 
program  management  support  to  minimize 
the  inclusion  of  stakeholders  who  don’t 
really  need  to  participate. 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

HW/SW 

The  identification  of 

HFEs  cannot  direct  the 

Changes: 

a  potential 

contractor  to  make  a 

performance  issue  or 

design  change 

Follow-up  with 

risk  in  the  current 

contractor 

design 

Additional  analysis  is  not 

after  problem 

in  the  current  contract  or 

identification 

A  need  for  design 
justification  needed 

within  scope 

from  the  contractor 

The  level  of  program 
office  support  to  question 

The  contractor 

the  contractor  or  ask  the 

requests  input  on 

contractor  to 

proposed  design 

reevaluate/redesign  the 

options 

system. 

A  capability  need 

The  willingness  of  the 

from  the  user 

contractor  to 

community  doesn’t 

reevaluate/redesign  the 

exist  in  the  design  or 
the  requirements,  and 

system. 

requires  a  change  to 

The  potential  cost  of  a 

the  current  HW/SW 
configuration. 

contractual  change 

A  need  to  review  the 

A  change  in  the 

design  in  detail  before  the 

mission  capabilities 

next  test  event  or 

technical  review 

A  need  to  control 
requirements  creep 


Judgments 


Challenges/  Difficulties 
Strategies/Mitigations 


Judge  whether  the  •  Determine  whether  the  design  feature  in  question 
issue/risk  is  a  is  within  scope  of  the  requirements, 

contractual  issue,  o  Strategy:  Consult  with  someone  who  knows 
a  human  the  scope  of  the  requirements  and  is  able  to 

performance  risk,  make  both  a  technical  and  contractual 

or  a  requirement  argument  for  the  contractor  to  resolve  the 

violation.  problem.  If  a  resolution  cannot  be  made  one 

on  one  with  the  contractor,  involve 

Judge  the  management  and  if  necessary,  the  contracting 

criticality  and  officer, 

significance  of  the 

problem  •  Improper  supporting  analyses  were  done  to 

generate  the  design.  The  contractor  can’t 
Judge  the  produce  a  substantive  analysis  that  defends  their 

adequacy  of  a  design.  This  causes  doubt  that  the  design  is 

proposed  redesign  compliant. 

to  address  the  o  Strategy:  Declare  the  design  is  noncompliant 

problem  and  determine  whether  or  not  it  fails  to  meet 

contractual  obligations.  Propose  (a)  that  an 
Judge  confidence  analysis  be  performed  to  substantiate  the 

in  the  contractor  design,  (b)  rework  the  design,  or  (c)  some 

other  contractual  actions  for  mitigation. 

Create  a  program  risk  if  deemed  necessary. 

•  The  contractor  is  uncooperative  and  thinks  the 
design  does  not  need  to  be  changed.  They  think 
they  are  right  and  the  Government  is  wrong, 
pushes  back  on  generating  alternatives, 
o  Strategy:  Request  further  substantiation  of 
their  position.  Find  the  right  requirements  to 
map  to  the  problem  found.  Seek  advice  from 
other  HFEs.  Contact  the  program  office 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties 
Strategies/Mitigations 

and/or  the  contractor’s  management.  Do  a 
comparison  of  the  system’ s  performance  as 
designed  to  the  required/needed  performance 
criteria  Generate  design  alternatives  using 
Government  resources  only. 

•  HFEs  identify  an  issue,  a  desire  to  resolve  it,  and 
a  need  for  time,  money  and  resources.  The 
program  office  does  not  agree  that  the  issue  is  a 
problem  worth  resolving,  and  disapproves  a 
request  for  additional  time,  money  and  resources 
for  an  analysis. 

o  Strategy:  Outline  the  concern  from  a 
scientific  standpoint.  Emphasize  the 
consequences  of  the  technical  risk  to  program 
cost  and  schedule.  Bring  in  supporting 
evidence  from  the  Fleet,  Safety  and  other 
stakeholders.  Get  management  support. 

Backing  down  from  the  issue  is  also  an 
option. 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties? 

HW/SW  Changes: 

A  design  deficiency 

A  need  to  meet  system 

Judge  the  necessity  of 

•  There  are  limited  Government  resources 

Gov’t  HFE(s) 

exists 

and  manpower 
requirements 

changing  the  design 

to  change  the  design, 
o  Strategy:  Either  add  more  resources 

(without 

An  immediate  HW/SW 

Judge  the  adequacy  of 

or  accept  the  risk  of  current  system 

contractor) 

need  from  the  Fleet 

A  change  in 

reusing  HW/SW  to 

performance  and  move  forward.  If 

explore/generate 
design  options  to 

Xhe  contractor  has 

requirements 

change  the  design 

the  system  can’t  be  redesigned,  wait 
and  see  if  the  concern  materializes  in 

later  present  to 

difficulty  translating 

A  desire  to  redesign  the 

Judge  whether  the 

actual  test. 

contractor,  Fleet, 

requirements  into 

system  to  have 

proposed  design  option 

IPX,  and/or 

design  options  (i.e. 

commonality  with 

is  viable  from  a  human 

•  The  program  office  objects  to  the  cost  of 

program  office 

(IPX  and 

needs  an  example  from 
the  Government) 

similar  systems 

Familiarity  with 

performance 

standpoint 

changing  the  design  (e.g.  contract 
modification,  revise  the  drawings, 
perform  additional  testing,  etc.). 

contractor  may  or 

Open  action  items  from 

technology  alternatives, 

Judge  if  the  redesign 

o  Strategy:  HFEs  prepare  a  human 

may  not 

a  previous  technical 

including  state  of  the 

options  are  more  cost 

performance  and/or  safety  focused 

simultaneously 

review  that  are 

art 

effective  than  the 

response  to  justity  the  cost  of 

investigate  other 

dependent  on  system 

original 

proposed  design(s)  and  emphasize 

design  options) 

redesign 

Lack  of  fimding  for  the 
current  contractor  to  do 
additional  analyses 

An  uncooperative 
response  from 
contractor  to  change  the 
existing  design 

Fleet  interest  in  specific 
technologies 

Using  lessons  learned 
from  previous 
experience  to  redesign 
the  system 

Judge  whether  the 
proposed  design  fits 
into  the  acquisition 
scheme,  if  the  work 
was  already  part  of  the 
contract,  and/or  the 
estimated  cost  for 
testing  and  integration 
is  feasible. 

Judge  the  reliability  of 
contractor  cost 

estimates 

why  it  is  worth  the  additional  cost. 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties? 

HW/SW  Changes: 

The  program  office 

The  complexity  of  the 

Judge  the  risk  of  having 

•  Some  IPT  members  are  uncooperative, 

did  not  approve  the 

design  change  (modify 

a  non-compliant  design 

have  conflicting  personalities 

IPX  (including 

design  options 

HW/SW  and 

o  Strategy:  Rely  on  good  IPT 

contractor) 

proposed  by  the 

operational/maintenance 

Judge  which  IPT 

leadership  to  manage  team 

explores/generates 

Government  HFEs 

procedure) 

members  and/or 

performance,  like  a  team  coach. 

design  options  to 

(see  previous 

stakeholders  need  to  be 

present  to  the 

decision  activity) 

Certain  elements  of  the 

involved  in  the  redesign 

•  The  system's  architecture  restricts 

program  office 

system  cannot  be  modified 

effort 

potential  design  changes  (e.g.  space 

No  current  design 

limitations,  computer  processing 

[Iterative  process] 

exists 

The  availability  of 

Judge  whether  the 

capacity,  etc) 

materials,  HW/SW  to 

design 

o  Strategy:  Consider  workarounds 

A  design  deficiency 

implement  the  proposed 

altematives/options  fit 

and  additional  design  options 

exists 

design  changes 

within  existing 

requirements  or  there  is 

•  The  logic  behind  a  unique  system 

Open  action  items 

Level  of 

a  need  to  changes  the 

requirement  is  questionable. 

from  a  previous 

cooperation/collaboration 

requirements. 

o  Strategy:  Consult  with  other  IPT 

technical  review 

between  IPT  members 

members  on  the  origin  and  the 

that  are  dependent 

Judge  the  impact  of 

logic  behind  the  requirement. 

on  system  redesign 

Using  lessons  learned  from 

redesign 

previous  experience  to 

implementation  on 

•  The  system  redesign  is  being  held 

The  contractor  has 

redesign  the  system 

program  cost  and 

accountable  to  the  original  system’ s 

difficulty 

schedule 

requirements,  some  of  which  may  no 

translating 

Availability  of  funds  to  do 

longer  apply. 

requirements  into 

the  redesign  work 

Judge  the  reliability  of 

o  Strategy:  Revise  the  requirements 

design  options  (i.e. 

the  contractor  cost 

needs  an  example 

Cost  to  implement  the 

estimate  for  system 

•  Some  IPT  members,  including  the 

from  the 

design  change 

integration  after 

contractor,  try  to  work  around  the 

Government) 

redesign 

Government  HFEs  by  getting  other 

people  to  do  the  work  and/or  going 

Program  redirection 

directly  to  the  users. 

o  Strategy:  Talk  to  the  IPT  members 
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Activities 

Cues 

What  triggers  this 
activity? 

Factors 

What  influences  this 
activity? 

Judgments 

Challenges/  Difficulties? 

and  try  to  reestablish  more 
effective  relationships.  Talk  to  the 
program  office  and  the  people  that 
they  were  going  to  and  make  them 
aware  of  the 

coordination/collaboration 
problem.  Inform  HFE  management 
and  let  them  decide  if  they  want  to 
pursue  the  violation  or  not. 
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APPENDIX  D  -  Cognitive  Activities  Mapped  to  Decision  Activities 


Examine/ 
Investigate 
initial 
manpower 
estimate  and 
the  assumed 
workload  for 
intended 
system 
design 

Create 
Program 
Risk  for 
Manpower 
(if  deemed 
necessary) 

Confirm 

workload 

and 

manpower 
for  new 
system 
design 

Evaluate 
initial 
system 
design  for 
potential 
problems 

E  olio  w- up 
with 

contractor 

after 

problem 

identification 

Gov’t  HEE(s) 
(without 
contractor) 
explore/generate 
design  options  to 
later  present  to 
contractor, 
Elect,  IPX, 
and/or  program 
office 

Entire  IPX  with 
contractor(s) 
explore/generate 
design  options  to 
present  to  the 
program  office 
[Iterative 
process] 

Sensemaking  and 
Situation  Assessment 

Risk  or  risk  mitigation 
strategy 

X 

X 

X 

X 

X 

Impact  to  Cost,  Schedule, 
or  Performance 

X 

X 

X 

X 

HFEs  with  Contractor, 

IPX,  Program  Office, 
Management 

X 

X 

X 

X 

X 

X 

X 

HFEs  with  Fleet,  User 
Community,  SMEs 

X 

X 

X 

X 

X 

X 

Technical  data  (Primarily 
HFE(s)) 

X 

X 

X 

X 

X 

X 

Analysis  results 

X 

X 

X 

X 

X 

X 

Technical  scope  of  work/ 
level  of  effort  for 
Contractor  or 

Government 

X 

X 

X 

X 
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Examine/ 
Investigate 
initial 
manpower 
estimate  and 
the  assumed 
workload  for 
intended 
system 
design 

Create 
Program 
Risk  for 
Manpower 
(if  deemed 
necessary) 

Confirm 

workload 

and 

manpower 
for  new 
system 
design 

Evaluate 
initial 
system 
design  for 
potential 
problems 

E  olio  w- up 
with 

contractor 

after 

problem 

identification 

Gov’t  HEE(s) 
(without 
contractor) 
explore/generate 
design  options  to 
later  present  to 
contractor, 
Elect,  IPT, 
and/or  program 
office 

Entire  IPT  with 
contractor(s) 
explore/generate 
design  options  to 
present  to  the 
program  office 
[Iterative 
process] 

Design  options 

X 

X 

X 

X 

Requirements 

X 

X 

X 

Leveraging  previous 
experience  and  lessons 
learned 

X 

X 

X 

X 

X 

X 

Literature  review,  prior 
studies,  prior  test  events 

X 

X 

X 

X 

X 

Planning/  Adapting/ 
Replanning 

For  collaboration/ 
coordination  with 
contractor,  IPT  members. 
Management,  other 
Stakeholders 

X 

X 

X 

X 

X 

Process  Used/Path 

Forward 

X 

X 

X 

X 

X 

X 

Technical  Scope  of  the 
Design 

X 

X 

X 

X 

X 

Uncertainty  and  Risk 
Management 

Validity  of  analysis 
method  and/or  analysis 
results 

X 

X 

X 

X 

Accuracy  and  validity  of 

X 

X 

X 

X 
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Examine/ 
Investigate 
initial 
manpower 
estimate  and 
the  assumed 
workload  for 
intended 
system 
design 

Create 
Program 
Risk  for 
Manpower 
(if  deemed 
necessary) 

Confirm 

workload 

and 

manpower 
for  new 
system 
design 

Evaluate 
initial 
system 
design  for 
potential 
problems 

E  olio  w- up 
with 

contractor 

after 

problem 

identification 

Gov’t  HEE(s) 
(without 
contractor) 
explore/generate 
design  options  to 
later  present  to 
contractor, 
Elect,  IPX, 
and/or  program 
office 

Entire  IPX  with 
contractor(s) 
explore/generate 
design  options  to 
present  to  the 
program  office 
[Iterative 
process] 

data  used  for  analysis 

Proposed  system 
attributes 

X 

X 

X 

X 

Process  to  Use/  Path 
Forward 

X 

X 

X 

X 

Scope  of  technical/ 
program  risks 

X 

X 

X 

Using  Opportunities 
and  Leverage  Points 

Coordinating/ 
Collaborating  on  Design 
Options 

X 

X 

X 

X 

X 

Requesting/  Generating/ 
Leveraging  data  and  data 
results 

X 

X 

X 

X 

X 

X 

X 

Requesting/  Generating/ 
Leveraging  Design 

Options 

X 

X 

X 
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APPENDIX  E  -  Summarized  Novice  Behaviors  and  Training  Needs 


Novice  Behaviors 

•  Lack  experience  and  instincts  (can’t  compare  current  work  to  past  projects) 

•  Lack  insight  and  good  judgment. 

•  Follow  procedure 

•  Tend  to  focus  only  on  the  task  at  hand.  Don’t  see  how  the  task  fits  in  with  the  rest  of  the 
design  process.  Don’t  see  design  complexity. 

•  Focus  on  whether  or  not  they  are  doing  to  task  correctly 

•  Ask  management  for  help  and  guidance  on  what  to  do 

•  Do  what  they  are  told 

•  Either  try  to  tackle  tasks  without  SME  inputs  or  network/collaborate  with  experts  to  access 
their  knowledge  base. 

•  Focus  primarily  on  design  requirements  compliance 

•  Eack  confidence  to  stand  their  ground  and  assert  their  position 

•  Novices  get  passionate/emotional/unreasonable 

•  Don’t  know  when  to  back  down 

•  Don’t  know  how  to  change  their  communication/presentation  approach  depending  on 
audience 

•  Don’t  understand  everyone  doesn’t  think  like  them 


Novice  Needs 


Knowledge  of: 

•  Available  analysis  methodologies  and  toolsets 

•  Policy 

•  Operations  and  Missions 

•  Which  stakeholders  to  involve  in  a  design  process 
Skills  in: 

•  Personal  interaction 

•  Communication/Presentation 

•  Negotiation 

How  to: 

•  Identify  and  define  risks,  mitigations  and  risk  exit  criteria 

•  Assess  the  impact  of  a  risk  or  design  on  cost,  schedule  or  performance 

•  Select  to  best  analysis  method  for  the  task  (the  one  that  will  produce  the  most  accurate  data 
results) 
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•  Assess  the  impaet  of  HSI  domain  data  and  data  analysis  results  on  other  funetions  (other  HSI 
domains,  HW/SW,  eost,  sehedule,  performanee,  eontraet  seope,  ete) 

•  Judge  aeeuraey  of  data,  data  results,  or  analysis  progress 

•  Judge  the  outeome  of  a  proeess/analysis 

•  Judge  the  adequaey  of  design  options 

•  Prediet  the  outeome  of  a  proeess/analysis 

•  Prediet  potential  problems 

•  See  the  “big  pieture” 

•  See  requirements  issues 

•  See  eontraetual  issues  related  to  teehnieal  issues 

•  Understand  that  eontraetors  will  push  baek  beeause  they  are  profit  oriented 

•  Reseope  tasking  (not  just  resehedule  tasking) 

•  Deviate  from  standards  and  still  be  eompliant 

•  Judge/Estimate  eost/sehedule/performanee  impaet 

•  Generate  design  options  (more  likely  to  ask  eontraetor  to  generate  them  or  seek  guidanee) 

•  Coordinate/Collaborate  with  stakeholders 
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